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. ABSTRACT 



The passage of the Carl D. Perkins Vocational and Applied 
Technology Education Amendments of 1998 (Perkins III) and the Workforce 
Investment Act of 1998 (WIA) marked a new era in the performance 
accountability partnership among the states, the U.S. Department of 
Education, and the U.S. Department of Labor (DOL) . Evaluation of vocational 
education programs ' performance is now characterized by equality management 
principles promoted by the Baldrige National Quality Program and the 
International Organization for Standardization. The following are among the 
key elements of the accountability process mandated by Perkins III and WIA: 
(1) identification of the customers and purposes for performance data; (2) 
the Office of Vocational and Adult Education’s Data Quality Initiative; (3) 
principles of outcome-based performance review; and (4) application of 
quality management principles to the Data Quality Initiative. A review of 
recent federal- and state-level evaluation initiatives reveals that 
outcome-based performance measurement systems consist of the following core 
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components: (1) a clear goal statement; (2) valid and reliable measurement; 

(3) standards adjusted for "relevant" differences in circumstances; and (4) 
performance-based consequences. Before undertaking the design of a 
high-quality performance measurement system, stakeholders must agree that 
relevant program activities and reliable performance measures can be linked 
in a practical way. (Contains 43 references.) (MN) 
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The Information Synthesis Project of the National Dissemination Center for Career and Tech- 
nical Education seeks to assist career and technical education practitioners and policymakers 
in choosing and using the best information available to apply to practice and to serve as an 
intermediary between local and expert knowledge. This paper is the second in the Information 
Paper series, which is designed to review and synthesize research by giving an overview of a 
distinct topic or a cross -disciplinary analysis of an emerging topic. 

David W. Stevens is a consultant to the current National Assessment of Vocational Education 
(NAVE) team and to the Program Reform Branch, Division of Vocational-Technical Educa- 
tion, Office of Vocational and Adult Education, U.S. Department of Education. He also man- 
ages a research alliance sponsored by the Office of Policy and Research, Employment and Train- 
ing Administration, U.S. Department of Labor. His contributions to previous NAVE work in- 
clude “Assessing the Impact of the Carl D. Perkins Vocational Education Act: Economic Devel- 
opment Issues” (1987) and “Occupations and Earnings of Former Vocational Education Stu- 
dents: Design Issues” (1992). He is the author of “Vocational Education Accountability in a 
Block Grant World” in Of Heart and Mind: Social Essays in Honor of Sar Levitan (Upjohn 
1996) and New Perspectives on Documenting Employment and Earnings Outcomes in Voca^ 
tional Education (NCRVE 1996). 
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lication: Michelle Kosmicki, Accountability Consultant, Nebraska Department of Education; 
Arlene Parisot, Director of Workforce Development, Montana Office of the Commissioner of 
Higher Education; Robert Sommers, Assistant Superintendent, Butler County (Ohio) Joint 
Vocational School District; and Robert Mahlman, Director of Assessment Services, Center on 
Education and Training for Employment, the Ohio State University. Susan Imel coordinated 
publication development and Sandra Kerka edited and formatted the manuscript. 
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Executive Summary 



The performance accountability partnership among the states, the U.S. Department of Educa- 
tion, and the U.S. Department of Labor entered a new era with the passage of the Carl D. 
Perkins Vocational and Applied Technology Education Amendments of 1998 (Perkins lllj and 
the Workforce Investment Act of 1998 (WlA). Quality management principles promoted by 
the Baldrige National Quality Program and the International Organization for Standardization 
characterize this new era. 

This paper addresses the application of these performance excellence criteria to the Perkins III 
performance measurement process. It begins with a description of Perkins III and WlA core 
indicators of performance, placing them in the historical context of 20th-century federal ac- 
countability for vocational-technical education in order to explain the current emphasis on 
compatibility among federal performance information systems. 

Following a review of major vocational education and work force development legislation and 
assessments, the heart of the paper is a discussion of Perkins III and WlA Title I and Title ll 
accountability processes, including- 

• Identification of the customers and purposes for performance data 

• The Office of Vocational and Adult Education’s Data Quality Initiative 

• Principles of outcome -based performance review 

• The application of quality management principles to the Data Quality Initiative 

An example of a state model for performance data collection is described. 

The paper concludes with a look at the future of accountability in career and technical educa- 
tion and work force development, touching on ways in which performance measurement prac- 
tices may differ in the 21st century. 
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Introduction 



A new generation of the performance accountability partnership among the states, the U.S. 
Department of Education, and the U.S. Department of Labor began with passage of the Carl D. 
Perkins Vocational and Applied Technology Education Amendments of 1998 (Perkins III) and 
the Workforce Investment Act of 1998 (WIA). Quality management principles promoted by 
the Baldrige National Quality Program (www.quality.nist.gov) and the International Organiza- 
tion for Standardization (www.iso.ch) characterize this new era. 

This paper begins with a description of the Perkins III and WIA core indicators of performance. 
These requirements are placed in historical context in chapter 3. Basic quality management 
principles are used in chapter 4 to identify opportunities to improve performance through man- 
agement adoption of refined measurement practices. Concluding observations about the future 
of education and work force development accountability are offered in chapter 5. 

Newcomers to career and technical education, work force development, and adult education 
accountability are introduced to similarities and differences in the federal core indicators that 
have been developed for Perkins III and WIA Title I (Workforce Investment Systems) and Title 
II (Adult Education and Literacy) . Those who are new to local and state performance account- 
ability responsibilities are challenged in chapters 4 and 5 to look beyond core indicators. Chap- 
ter 4 highlights the federal-state partnership that is identifying exemplary state performance 
measurement practices and investing in forums to bring these innovations to the attention of 
other state and local colleagues. 

A balanced treatment of secondary and higher education performance indicators and issues is 
offered here. The evolution of Tech Prep from a basic 2+2 articulation of high school and 
community college coursework into diverse approaches including middle school and college 
components has increased the importance of aligning secondary and higher education perfor- 
mance indicators and the information systems that produce them. 

A basic theme throughout this paper is that public release of core indicator information for 
Perkins III and WIA Title I and Title II will increase public and media interest in other aspects 
of program performance. Many of today’s management information systems do not contain the 
data needed to respond to the expected questions. Other information systems do contain the 
necessary data, but are not designed to make this information available quickly and at low cost. 

Quality Management Principles 



Attainment of a high level of performance quality requires a high standard of management 
quality. This, in turn, requires high-quality performance measurement. An accurate under- 
standing of today’s performance is needed to pursue future improvement. 

The International Organization for Standardization (ISO) Principles 

Two of the ISO principles for assessing quality management standards are highlighted here and 
in chapter 4. 
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1 . Continual improvement — employing a consistent organization-wide approach to continual 
improvement of the organization’s performance; providing people with training in the methods 
and tools of continual improvement; making continual improvement of products, processes, 
and systems an objective for every individual in the organization; establishing goals to guide 
and measures to track continual improvement; and recognizing and acknowledging im- . 
provements. 

2. A factual approach to decision making — ensuring the data and information are sufficient- 
ly accurate and reliable; making data accessible to those who need it; analyzing data and 
information using valid methods; and making decisions and taking action based on factual 
analysis, balanced with experience and intuition. 

The remaining six ISO quality management principles are customer focus, leadership, involve- 
ment of people, process approach, system approach to management, and mutually beneficial 
supplier relationships. Each of the eight principles is used in chapter 4 as a framework for an 
assessment of the federal-state performance measurement partnership under Perkins III. 

The Baldrige National Quality Program Education Criteria 

The Education Criteria for Performance Excellence volume released by the Baldrige National 
Quality Program (US. Department of Commerce 2000) contains seven criteria. Two of these 
are of particular interest here (ibid., p. 27): 

1. Information and analysis — examining an organization’s performance measurement system 
and how the organization analyzes performance data and information. This includes the 
selection of indicators and evidence of their use and effectiveness in daily operations, infor- 
mation reliability, an understanding of improvement options, projections of data to support 
planning, and steps taken to keep the performance measurement system current with ser- 
vice needs and directions. 

2. Student performance results — segmented by student groups as appropriate, and including 
appropriate data relative to comparable organizations and student populations. The ques- 
tion to be answered is: What are your current levels and trends in key measures and/or 
indicators of student performance? 

The remaining five Baldrige education criteria for judging performance excellence are leader- 
ship, strategic planning, student and stakeholder focus, faculty and staff focus, and educational 
and support process management. 

The Baldrige criteria complement the ISO quality management principles, as both are used in 
chapter 4 to assess the federal-state performance measurement partnership under Perkins III. 
A related document has just been released: Measuring Quality: Choosing among Surveys and 
Other Assessments of College Quality (Borden and Owens 2001) . 
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Two preliminary steps must be taken before the Baldrige and ISO performance excellence crite- 
ria are applied to the Perkins III performance measurement process in chapter 4. The core 
indicators of performance for Perkins III and WIA Title I and Title II are introduced with 
limited commentary in the next chapter. Historical perspectives on accountability processes in 
career and technical education and work force development follow. 


Introduction 


o ■ 






Perkins III and WIA Title I and Title II 
Core Indicators of Performance 



A logical first step in an examination of 21st-century accountability issues is to document what 
is required today. That is the topic covered in this chapter. These core indicator requirements 
are placed in historical perspective in the next chapter. The content of these two chapters is 
needed as background for understanding the quality management interpretation that is given 
to the Perkins III federal-state accountability partnership in chapter 4. 

The Perkins III core indicators are covered first, followed by the WIA Title I measures and then 
the WIA Title II core indicators. Each set of indicators is prefaced by a brief description of the 
performance accountability goal found in each of the laws. 

Perkins III 

Perkins III, Section 2, states the purpose of the act: 

The purpose of this Act is to develop more fully the academic, vocational, and tech- 
nical skills of secondary students and postsecondary students who elect to enroll in 
vocational and technical education programs. 

The purpose of the act is to be achieved by — 

(1) building on the efforts of States and localities to develop challenging academic 
standards; (2) promoting the development of services and activities that integrate 
academic, vocational, and technical instruction, and that link secondary and post- 
secondary education for participating vocational and technical education students; 

(3) increasing State and local flexibility in providing services and activities designed 
to develop, implement, and improve vocational and technical education, including 
tech-prep education; and (4) disseminating national research, and providing profes- 
sional development and technical assistance, that will improve vocational and tech- 
nical education programs, services, and activities. 

Two sections of Perkins III contain the performance accountability expectations of the 1998 
amendments. Section 1 13 defines state performance accountability requirements and opportu- 
nities. Section 1 14 describes national performance reporting responsibilities. 

State Responsibilities (Section 113) 

The goal for state performance accountability is — 

to assess the effectiveness of the State in achieving statewide progress in vocational 
and technical education, and to optimize the return of investment of Federal funds 
in vocational and technical education activities. (Section 113(a)) 
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Section 113(a) begins: 



The purpose of this section is to establish a State performance accountability system, 
comprised of the activities described in this section. 

Section 113(b)(2)(C) offers an alternative approach: 

If a State previously has developed State performance measures that meet the re- 
quirements of this section, the State may use such performance measures to measure 
the progress of vocational and technical education students. 

Three basic activities comprise a state’s performance accountability system: 

1. Core indicators of performance, which are described in Section 1 13(b) (2) (A) 

2. Any additional, or discretionary, indicators of performance that might be identified 

3. A state adjusted level of performance for each of the core indicators and a state level of 
performance for each additional indicator that has been identified 

The Core Indicators of Performance (Section 113(b)(2)(A)) 

The minimum measures that each state must identify to satisfy the core indicators*of perfor- 
mance requirement are as follows: 

1. Student attainment of challenging state-established academic and vocational and tech- 
nical, skill proficiencies 

2. Student attainment of a secondary school diploma or its recognized equivalent, a profi- 
ciency credential in conjunction with a secondary school diploma, or a postsecondary 
degree or credential 

3. Placement in, retention in, and completion of, postsecondary education or advanced 
training, placement in military service, or placement or retention in employment 

4. Student participation in and completion of vocational and technical education programs 
that lead to nontraditional training and employment 

The Office of Vocational and Adult Education (OVAE) in the U.S. Department of Education 
has translated these 4 core indicators of performance into 14 subindicators, which were re- 
leased in a January 2000 document titled Core Indicator Framework. More information about 
this framework and the subindicators can be found in chapter 4. 

& 

Additional Indicators of Performance (Section 113(b)(2)(B)) 

Each state is offered an opportunity to broaden the portfolio of performance indicators beyond 
the required core indicators described here. The core indicators of performance in Section 




Core 

Indicators 



113(b)(2)(A) are described as minimal required evidence of progress. A 
supplement the core indicator evidence with additional information. 



state may choose to 



“Progress” and the Levels of Performance Issue 
(Section 113(b)(3)(A)(l) and (v!)) 



Section 1 13(b) (3) (A) (i) defines state responsibilities for selection of a target performance level 
for each of the core indicators: 



The levels of performance established under this subparagraph shall, at a minimum — 

(I) be expressed in a percentage or numerical form, so as to be objective, quantifi- 
able, and measurable; and (II) require the State to continually make progress toward 
improving the performance of vocational and technical education students. 

Measurement of progress requires the definition of a start time and an end time and the calcu- 
lation of indicator values for each of these times. Typically, these are not independent decisions. 

Section 1 13 (b) (3) (A) (vi) provides an incentive for states to exchange information: 

The agreement [between a State and the Secretary of Education] shall take into 
account — (I) how the levels of performance involved compare with the State ad- 
justed levels of performance established for other States taking into account factors 
including the characteristics of participants when the participants entered the pro- 
gram and the services or instruction to be provided; and (II) the extent to which 
such levels of performance promote continuous improvement on the indicators of 
performance by such State. 

Each annual cycle of state performance accountability can be thought of as climbing through 
progressively higher base camps in preparation for a final successful assault on a mountain top, 
except the continuous progress requirement assumes that the altitude of the peak is rising — 
one can never expect to reach the top. There will always be a new challenge to pursue. 

State Reporting (Section 113(c)(1)) 

Each state is required to — 

annually prepare and submit to the Secretary [of Education] a report regarding — 

(A) the progress of the State in achieving the State adjusted levels of performance 
on the core indicators of performance; and (B) information on the levels of perfor- 
mance achieved by the State with respect to the additional indicators of performance, 
including the levels of performance for special populations. 

The core indicators of performance, expressed in a percentage or numerical form, and special 
population performance information must be included in a state’s annual report, as must addi- 
tional indicators if they have been included in a state’s plan in accordance with Section 
113(b)(2)(B). 
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Indicators 



State Responsibilities: A Summary Statement 



This concludes an overview of state responsibilities and opportunities for Perkins 111 perfor- 
mance accountability reporting, as these are defined in Section 113 of the act. Section 113 
offers wide latitude to use existing state performance accountability systems if these satisfy core 
indicator reporting requirements. 

Federal Responsibilities (Section H4) 

Section 114(a)(1) and (2) define the Secretary’s basic responsibilities for Perkins III perfor- 
mance accountability reporting: 

(1) The Secretary shall collect performance information about, and report on, the 
condition of vocational and technical education and on the effectiveness of State 
and local programs, services, and activities carried out under this title in order to 
provide the Secretary and Congress, as well as Federal, State, local, and tribal agen- 
cies, with information relevant to improvement in the quality and effectiveness of 
vocational and technical education. The Secretary annually shall report to Congress 
on the Secretary’s aggregate analysis of performance information collected each year 
pursuant to this title, including an analysis of performance data regarding special 
populations. 

(2) Compatibility — ^The Secretary shall, to the extent feasible, ensure that the per- 
formance information system is compatible with other Federal information systems. 

The Secretary is obliged to collect information that is relevant to improvement, must perform 
an aggregate analysis of performance information collected, and is instructed to strive for com- 
patibility with other federal information systems to the extent feasible. Prominent among these 
other federal information systems are WIA Title I and Title II performance indicator systems. 

IIWIATitlel 

The stated purpose of the Workforce Investment Act of 1998, Title I, Workforce Investment 
Systems, is found in Section 106: 

The purpose of this subtitle is to provide workforce investment activities, through 
statewide and local workforce investment systems, that increase the employment, 
retention, and earnings of participants, and increase occupational skill attainment 
by participants, and, as a result, improve the quality of the workforce, reduce welfare 
dependency, and enhance the productivity and competitiveness of the Nation. 

Services Offered 

• The services to achieve the act’s purpose are defined in Section 134 (d) (1) . Three basic catego- 
ries of service are available to eligible adults and dislocated workers — core services, intensive 
services, and training services. 
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1. Core Services. These include eligibility determination; outreach, intake, and orientation to 
the information and other services available through the one-stop delivery system; ini tial 
assessment of skill levels, aptitudes, abilities, and supportive service needs; job search and 
placement assistance: and where appropriate, career counseling. 



Core 

Indicators 



2. Intensive Services. These include comprehensive and specialized assessments of the skill 
levels and service needs of adults and dislocated workers; development of an individual 
employment plan; group counseling; individual counseling and career planning; case man- 
agement for participants seeking training services; and short-term prevocational services, 
including development of learning skills, communication skills, interviewing skills, punctu- 
ality, personal maintenance skills, and professional conduct, to prepare individuals for 
unsubsidized employment or training. 

3. Training Services. These may include occupational skills training, including training for 
nontraditional employment; on-the-job training; programs that combine workplace train- 
ing with related instruction, which may include cooperative education programs; training 
programs operated by the private sector; skill upgrading and retraining; entrepreneurial train- 
ing; job readiness training; adult education and literacy activities provided in combination 
with services described above; and customized training conducted with a commitment by an 
employer or group of employers to employ an individual upon successful completion of the 
training. 



Performance Accountability (Section 136) 



The stated purpose of the WIA Title I performance accountability system is — 

to establish a comprehensive performance accountability system, comprised of the 
activities described in this section, to assess the effectiveness of States and local areas 
in achieving continuous improvement of workforce investment activities funded under 
this subtitle, in order to optimize the return on investment of Federal funds in state- 
wide and local workforce investment activities. 

Section 136(b) (1) requires that state performance measures consist of — 



• Core indicators, which are defined in Section 136(b) (2) (A) 

• A customer satisfaction indicator, which is defined in Section 136(b) (2) (B) 

• Additional indicators of performance (if any) identified by the state 

• A state-adjusted level of performance for each core indicator 



The Title! Core Indicators of Performance (Section 136(b)(2)(A)) 

There are 17 Title I core measures of performance: 

• Adult entered employment rate 

• Adult employment retention rate 

• Adult earnings change 

O 
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Adult employment and credential rate 
Dislocated worker entered employment rate 
Dislocated worker employment retention rate 
Dislocated worker earnings replacement rate 
Dislocated worker employment and credential rate 

Older youth (eligible youth aged 19-21 at registration) entered employment rate 
Older youth employment retention rate 
Older youth earnings gain rate 
Older youth credential rate 

Younger youth (eligible youth aged 14-18 at registration) skill attainment rate (not an 
“exit" measure) 

Younger youth diploma or equivalent attainment rate 
Younger youth retention rate 

Customer satisfaction — two indices, calculated from survey data collected from partici- 
pants and employers (who participated in On-the-Job Training, Rapid Response, or La- 
bor Exchange activities) 

WIA subpopulation definitions require careful attention. Section 101 of the act defines “adult” 
as an individual who is age 18 or older, with two exceptions for allocation of federal funds to the 
states: 

1. Section 127(b)(2) defines a “disadvantaged youth” as an individual age 16 through 21 
who received an income, or is a member of a family that received a total family income, 
that, in relation to family size, does not exceed the higher of — (i) the poverty line; or (ii) 
70 percent of the lower living standard income level. 

2. Section 132(b)(v) defines “adult” as an individual who is not less than 22 and not more 
than age 72. 

Section 101 defines an “eligible youth” as an individual who is not less than age 14 and not 
more than age 21; is a low- income individual; and is an individual who is one or more of the 
following: deficient in basic literacy skills; a high school dropout; homeless, a runaway, or a 
foster child; pregnant or a parent; an offender; or an individual who requires additional assis- 
tance to complete an educational program, or to secure and hold employment. This definition 
does not apply to subtitle C (Job Corps) and subtitle D (National Programs). 

II WIA Title II 



Core 

Indicators 




The Workforce Investment Act of 1998, Title II, Adult Education and Literacy, statement of 
purpose is — 

to create a partnership among the Federal Government, States, and localities to 
provide, on a voluntary basis, adult education and literacy services, in order to— (1) 
assist adults to become literate and obtain the knowledge and skills necessary for 
employment and self-sufficiency; (2) assist adults who are parents to obtain the edu- 
cational skills necessary to become full partners in the educational development of 
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their children; and (3) assist adults in the coiripletion of a secondary school educa- 
tion. 

Services Offered 

The definition of “adult education” is services or instruction below the postsecondary level for 
individuals — (A) who have attained 16 years of age; (B) are not enrolled or required to be 
enrolled in secondary school under state law; and (C) who — (i) lack sufficient mastery of basic 
educational skills to enable the individuals to function effectively in society; (ii) do not have a 
secondary school diploma or its recognized equivalent, and have not achieved an equivalent 
level of education; or (iii) are unable to speak, read, or write the English language. 

Performance Accountability 

The stated purpose of the Title II performance accountability system, in Section 212, is — 

to establish a comprehensive performance accountability system, comprised of the 
activities described in this section, to assess the effectiveness of eligible agencies in 
achieving continuous improvement of adult education and literacy activities funded 
under this subtitle, in order to optimize the return on investment of Federal funds in 
adult education and literacy activities. 

The eligible agency performance measures are core indicators of performance, additional indi- 
cators of performance (if any) identified by the eligible agency, and an eligible agency adjusted 
level of performance for each core indicator. 

The Title ii Core Indicators of Performance 
(Section 212(b)(2)(A)) 

The core indicators shall include the following: 

• Demonstrated improvements in literacy skill levels in reading, writing, and speaking the 
English language, numeracy, problem solving, English language acquisition, and other lit- 
eracy skills 

• Placement in, retention in, or completion of postsecondary education, training, unsubsidized 
employment, or career advancement 

• Receipt of a secondary school diploma or its recognized equivalent 

For each eligible agency submitting a state plan, there shall be established levels of performance 
for each of the core indicators of performance. The levels of performance shall, at a minimum, 
be expressed in objective, quantifiable, and measurable form and show the progress of the eli- 
gible agency toward continuously improving in performance. 
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Compatibility of Perkins III and WIA Title I and 
il Title II Performance Information Systems 

Before introducing an historical perspective on vocational and technical education arid work 
force development accountability in the next chapter, it is important to consider why Perkins 
III, Section 114(a)(2), requires the Secretary of Education to ensure, to the extent feasible, 
that the Perkins III performance information system is compatible with other federal informa- 
tion systems. One possible reason is to limit each state’s reporting burden and costs. Another 
possible reason is to permit comparisons to be made among state programs that receive federal 
funds under Perkins III and WIA Title I and Title II. 

If the basic reason for seeking performance information system compatiblity is state and federal 
cost containment, it is important to consider what information of strategic management value 
might be lost in pursuing compatibility. If the fundamental reason for seeking performance 
information system compatibility is to enable a comparison of performance across federally 
funded programs, it is important to consider the objective of such comparisons. 

The purpose of Perkins III is “to develop more fully the academic, vocational, and technical 
skills of secondary students and postsecondary students who elect to enroll in vocational and 
technical programs.” 

The purpose of WIA Title I is “to provide workforce investment activities... that increase the 
employment, retention, and earnings of participants, and increase occupational skill attain- 
ment by participants, and, as a result, improve the quality of the workforce, reduce welfare 
dependency, and enhance the productivity and competitiveness of the Nation.” 

The purpose of WIA Title II is “to assist adults to become literate and obtain the knowledge 
and skills necessary for employment and self-sufficiency... and to become full partners in the 
educational development of their children.” 

The stated purposes of Perkins III and WIA Title I and Title II complement each other. The 
circumstances of the subpopulations served are quite different at the time of participation in 
one of the federally funded activities. The immediate program objectives are strikingly differ- 
ent, too. These conclusions suggest that pursuit of a federal capability to compare performance 
across Perkins III and WIA Title I and Title II is not a basic motivation for seeking compatibil- 
ity of performance accountability systems across programs. This leaves state cost containment 
as a likely candidate for the source of the Perkins III, Section 114(a) (2) requirement that the 
Secretary of Education ensure compatibility to the extent feasible. 

This chapter introduces the Perkins III and WIA Title I and Title II accountability goals and 
core indicators of performance. Chapter 3 explores whether there was a trend toward more, or 
less, rigorous federal performance measurement requirements leading up to Perkins III and 
WIA. 
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Historical Perspective 



This chapter seeks to provide a basic understanding of 20th-century federal accountability for 
vocational and technical education and work force development programs. This background 
information is expected to be useful for those who may have opportunities to influence the next 
cycle of 21st-century performance reporting, including continuing implementation of the core 
indicators described in chapter 2. Only views about accountability issues are included here. 
Readers interested in vocational education history per se are referred to Lazerson and Grubb 
(1974) and Giordano-Evans (1975). 

Pre-1960 



The historical coverage begins with excerpts from a paper co-authored by Rupert N. Evans and 
Paul Violas (1983): 

The early history of proposals for vocational education is full of expected outcomes 
for such programs. Rarely, however, were there proposals that these outcomes be 
tested, let alone suggestions as to how they should be tested... .Almost all of the 
emphasis was on assessing process rather than outcome, (p. 33) 

Not until vocational educators began to be trained as researchers in the 1950s was 
there an emphasis within vocational education on testable outcomes. Even today 
[1983], most of the emphasis on experimentation comes from outside the field, (p. 

34) 

One problem with the task [of assessing evidence supporting or rejecting particular 
outcomes] is that the list of outcomes we were given represents a view of the evalu- 
ation of social programs that has not always been held by many people, especially in 
the early days of vocational education. This view holds that it is desirable to specify 
in advance the outcome (s) expected of a program in order to determine more easily 
the extent to which these outcomes have been achieved, (p. 34) 

In contrast, the advocates of vocational education have tended to identify problems 
of society and of individuals and then to specify which of these problems they felt 
vocational education could solve, (p. 34) 

If those who plan and implement an activity seek different outcomes from those who 
evaluate it, one should not be surprised if the evaluation is negative, (p. 35) 

A second major problem is the assumption that all of vocational education does or 
should have the same outcomes.. ..Now it seems to be assumed by nOnvocational 
educators that all should produce employees, (p. 35) 

A third major problem is that our list of eighteen outcomes was developed in 1982 by 
a group of researchers and policy makers. It clearly does not include some of the 
important outcomes desired.. .by practitioners, who are more concerned with main- 
tenance or expansion of enrollments and the solution of other daily problems than 
with grand designs, (p. 35) 
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It would appear that both defenders and critics were reasonably happy to make as- 
sumptions about outcomes based on logic, or exhortations, rather than to test out- 
comes with data. (p. 36) 

From 1917 until 1963 [Vocational Education Bulletin #1] was the “Bible” for voca- 
tional education programs that received federal funds. It contains absolutely no ref- 
erence to outcomes other than expenditure of funds for purposes allowed by federal 
law. (p. 45) 

It is our view that the assessment of outcomes in order to improve individual pro- 
gram decisions is much more desirable and feasible than assessment of a group of 
disparate programs serving disparate communities and disparate groups of individu- 
als under disparate economic conditions, (p. 50) 



The views expressed by these clearly frustrated long-time vocational education practitioners 
can be summarized in the form of three questions: 



1 . Should a single list of core indicators be defined for sweeping performance accountability 
covering the entire set of “vocational and technical education” activities? Twenty years ago, 
Evans and Violas answered “no.” The Perkins III core indicators of performance reflect a 
qualified “yes.” 

2. If one believes that a single list of core indicators of performance can be defined, is there a 
practical way to measure these outcomes to the mutual satisfaction of varied constituencies 
with different standards of indicator validity and reliability? 

3. If doubt is harbored about a practical way to measure outcomes to the mutual satisfaction of 
varied constituencies, what performance accountability consequences might be expected to 
flow from sustained disagreement about how to measure statewide progress in vocational 
and technical education? 

II The 1960s and 1970s 



o 
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The Vocational Education Act of 1963 and the Vocational Education Amendments of 1968 
introduced fundamental changes in federal funding criteria and targeted populations. The Edu- 
cation Amendments of 1976 contained the first midcentury accountability language. 

The 1976 Amendments charged the National Institute of Education with conducting “an analysis 
of the means of assessing program quality and effectiveness” (National Institute of Education 
1979, p. v). Unfortunately, the Institute’s Final Report (1981) warned that — 

the results reported do not constitute, and should not be read as, an assessment of 
the effectiveness of either secondary or postsecondary vocational education pro- 
grams. They are too limited — ^by both data available for research and the difficulty of 
the research problem — to attribute outcomes, both economic and noneconomic, to 
particular educational experiences, (p. VII-22) 





The 1976 Amendments, Section 164(b)(4)(A), also charged the National Advisory Council 
on Vocational Education to — 
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identify, after consultation with the National Commission for Manpower Policy, the 
vocational education and employment and training needs of the nation and assess 
the extent to which vocational education, employment training, vocational rehabili- 
tation, and other programs under this and related Acts represent a consistent, inte- 
grated and coordinated approach to meeting such needs, (p. vii) 

The challenge to determine whether federal legislation represented a consistent, integrated, 
and coordinated approach soon became more difficult with passage of the Comprehensive 
Employment and Training Act Amendments of 1978. The Comprehensive Employment and 
Training Act of 1973 (CETA) had — 

consolidated under one legislative umbrella a large number of. . .previously fragmented 
and specialized efforts and placed primary reliance instead on local levels of govern- 
ment to plan and deliver a more comprehensive set of services. In other words, CETA 
moved toward decategorization of programs and decentralization of responsibility. 
Currently, the entire country is divided into 451 “prime sponsorships” (jurisdictions 
with 100,000 population or more) who have first- line responsibility for operating the 
program. (National Commission for Manpower Policy 1978, p. 3) 

The CETA Amendments of 1978 gave the local prime sponsors control of federal funds to be 
invested in employment training of economically disadvantaged youths still in school. No pre- 
sumptive provider of these training services appeared in the amendments, but a prime sponsor 
was required to use local organizations of “demonstrated effectiveness.” 



The CETA Amendments of 1978 offered no criteria for assessing a local organization’s “demon- 
strated effectiveness.” Furthermore, the CETA funds flowed directly to the local prime spon- 
sors, whereas federal vocational education funds flowed through states to local areas. Limited 
availability of reliable performance indicators for local vocational education programs opened 
the door to widespread ignoring of local vocational education programs by the prime sponsors 
(Stevens 1979). 

The block grant era, looked upon with nostalgia by Evans and Violas in 1983, had given way to 
a flow of federal vocational education funds based largely on concentrations of poverty and an 
explicit goal to increase earnings. Anecdotal evidence at the time indicated that many voca- 
tional educators were concerned about the CETA youth programs because of perceived low 
standards for program completion and certification of competence. 
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The Carl D. Perkins Vocational education Act 
of 1984 (PL 98-524) 

Section 403 of the 1984 act mandated a National Assessment of Vocational Education. The 
Final Report (Wirt et al. 1989) Executive Summary begins: 

The National Assessment of Vocational Education (NAVE) has studied the imple- 
mentation of the Carl D. Perkins Act of 1984 and the status of vocational education 
at the secondary and postsecondary levels. We conclude that the basic goals of in- 
creasing the access of special populations to high-quality vocational education and 
improving the overall quality of programs are sound, but that the legislation is a 
weak instrument for achieving these goals, (p. vii) 

The Final Report includes separate secondary and postsecondary policy recommendations for 
performance indicator improvements. 

Secondary Performance indicators 

Each state would develop performance indicators to measure the success of voca- 
tional education for different populations of students and to achieve reform. The 
indicators would include information on academic achievement, vocational attain- 
ment and occupational skills, employment outcomes, and the continuity of student 
training between secondary and postsecondary levels; States would report on stu- 
dent performance within two years of reauthorization, (ibid., p. xv) 

The NAVE Final Report authors, John G. Wirt, Lana D. Muraskin, David A. Goodwin, and 
Robert H. Meyer, proposed that — 

the last component of the secondary policy should be to test alternative ways of 
linking performance and improvement through allocating resources on the basis of 
information from indicators, (p. xvi) 




Postsecondary Performance Indicators 

States would develop indicators of performance of postsecondary institutions. These indicators 
would measure the performance of postsecondary vocational institutions in three main areas: 

1. Labor Market Outcomes, including the rates at which students are placed in jobs, whether 
the placements are ‘training related,’ the duration of employment and unemployment, and 
the level of earnings at job entry and selected times thereafter. 



2. Learning Outcomes, including the rates at which students attain state certification (in 
fields with certification), achieve occupational competencies, and improve their scores on 
tests of academic knowledge. 





3. Educational Attainment, including the rates at which students earn degrees and certifi- 
cates, takes courses in a sequence, and enroll in more advanced level courses such as in- 
tended by Tech Prep or other similar programs, (pp. xix and xx) 

The NAVE recommendation for performance funding at the postsecondary level is more direct 
than the proposal to “test alternative ways of linking performance and improvement through 
allocating resources on the basis of information from indicators” at the secondary level. 

States would distribute funds from the Perkins Act according to the performance of 
institutions as measured by the indicators developed. Institutions with excellent per- 
formance would receive higher rewards, (p. xx) 

redera! Leadership in tducationai Reform 

The NAVE defined the federal role with respect to vocational education indicators. The vision 
expressed in this 1989 Final Report is consistent with the overall theme of 21st-century ac- 
countability: 

The performance indicators to be developed by the states are intended to encourage 
reform. The information derived from the indicators is intended to create incentives 
for reform by provoking questions from vocational education officials, legislators, 
governors, teachers, students, parents, employers, and others about what vocational 
education is accomplishing and what it should accomplish. The information will also 
assist state officials in setting goals and direction for the improvement of vocational 
education. A critical role for the federal office will be to provide guidance to the 
states on measurement priorities and expert technical assistance in indicator devel- 
opment. (p. xxi) 

The CariD. Perkins Vocationai and Appiied Technoiogy 
Education Act of 1990 (P.L 101-392) 

The language of the 1990 act reflected the 1989 NAVE recommendations. Section 115(a) 
required that — 

each state board receiving funds under this Act shall develop and implement a state- 
wide system of core standards and measures of performance for secondary and post- 
secondary vocational education programs. 

Section 1 15 (c) further required that — 

in developing the standards and measures included in a system developed under 
subsection (a), the State board shall take into consideration — (1) standards and 
measures developed under job opportunities and basic skills training programs estab- 
lished and operated under a plan approved by the Secretary of Health and Human 
Services. . .and (2) standards prescribed by the Secretary of Labor under section 106 
of the Job Training Partnership Act. 
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The Job Training Partnership Act of 1982 OTPA) was the successor to the CETA Amend- 
ments of 1978 and the predecessor to the Workforce Investment Act of 1998. The Job Oppor- 
tunities in the Business Sector (JOBS) initiative targeted welfare recipients who needed basic 
skills training to enter the work force. 



The 1990 Vocational and Applied Technology Act phrase “shall take into consideration” was 
subject to different interpretations. Advocates of movement toward a common set of post- 
program outcome measures interpreted this to mean that a state board was obliged to take the 
necessary steps to adopt the same measures that were used by their Job Training Partnership 
Act and Family Support Act counterparts. Those who opposed consolidation interpreted the 
phrase to mean that a state board was required only to consider a joining of forces, but that it 
retained full authority to reject such a move. 



Regardless of how the specific phrasing was interpreted, there was a clear desire to limit a state’s 
data collection burden, while promoting the collection of data that would support inquiries 
across program boundaries. In other words, there was increasing interest in monitoring partici- 
pant flows among different education and employment training programs through time (Stevens 
1985, 1991). 



The 1990 act. Section 115(b)(2)(a) through (d), required a state’s system of core measures of 
performance to include one or more measures of performance selected from four basic catego- 
ries of student attainment: 



• Competency attainment 

• Job or work skill attainment or enhancement, including progress in achieving occupa- 
tional skills 

• Retention in school or completion of secondary school or its equivalent 

• Placement into additional training or education, military service, or employment 

The 1990 act further required each state board to provide incentives or adjustments that would 

encourage service to targeted groups or special populations. 

The federal mandate to select one or more measures of performance could be satisfied without 

much regard for the spirit of the 1989 NAVE recommendation: 

• Competency attainment could be measured by completion of a vocational or academic 
course. 

• Work skill attainment or enhancement could be documented by completion of one or 
more vocational courses, short of program completion. 

• Retention in school or completion of secondary school or its equivalent could be satisfied 
by promotion to a higher grade or award of a General Educational Development (GED) 
certificate. 
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• Placement into additional training or education, military service, or employment could 
be recorded by a former student’s presence in postsecondary education, military service, 
or employment, without any other descriptors of the quality of the transition destination 
from vocational education. 
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No threat of sanction or use of a federal funds “hammer” emerged from the 1990 act. There was 
simply a common understanding that the proliferation of federal funding of training programs 
through departments other than the U.S. Department of Education increasingly loomed as a 
threat to continued funding of vocational and technical education. 



There was general federal and state agreement in the vocational and technical education com- 
munity that there was some urgency to be able to provide Congress and the public with reliable 
performance information. However, at a 1991 Design Conference for the National Assessment 
of Vocational Education, Norton Grubb (1992) warned that — 

The Perkins Act requires state committees of practitioners to adopt performance 
measures by August 1992. It will therefore be possible to determine what these mea- 
sures are in late 1992. However, since the Perkins Act requires only that measures be 
developed (not implemented) and are silent about how such measures are to be 
used, the real effects of performance measures on data collected, state use of perfor- 
mance measures, and reforms undertaken in response to performance measures will 
not be clear until later, perhaps 1993 or (realistically) 1994 or 1995. (p. 10) 

The new National Assessment of Vocational Education Interim Report to Congress (U.S. De- 
partment of Education 1994) included the following findings about state implementation of the 
1990 act’s performance accountability requirements: 

In the spring of 1992, most states were still in the process of developing their plans. 

Many had not yet decided which measures to use, nor had they set standards for 
performance, (p. 285) 

More than two-thirds of secondary and postsecondary state agencies were develop- 
ing new performance measures, rather than simply adjusting or recycling measures 
that had already been in use. (pp. 286-287) 

With little or no state guidance, most school districts and postsecondary institutions 
seem to be waiting to see how their state’s plan will affect them. (p. 288) 

Most states intend to use a far greater number and variety of measures than the 
minimum required in the Act. (p. 292) 

At least one-half of the states will apply their performance-based systems of account- 
ability to all students taking vocational, occupational, or technical courses, (p. 297) 

Many of the data that were collected by schools were not reported to states, (p. 300) 
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It will be important to examine the reliability and validity of new instruments devel- 
oped specifically for Perkins Act accountability systems. Poor-quality measures could 
undermine the utility of the whole system, (p. 302) 

It will be much more difficult to achieve common measures once state systems are 
adopted and operational, (p. 302) 




The Carl D. Perkins Vocational and Applied Technology 
education Act Amendments of 1998 (P.L. 105-332) 

The 1998 Amendments authorized another National Assessment of Vocational Education. 
The research team is completing its analysis in anticipation of a mid-2002 report to Congress. 

II rurther Reading from the 1980s and 1990s 



The 1982 Job Training Partnership Act and 1990 Perkins Vocational and Applied Technology 
Education Act focused attention on state performance accountability practices and opportuni- 
ties. Selected studies are identified here, spanning 12 years between 1987 and 1998, for readers 
who seek more detailed historical perspective, particularly about the employment indicator. 

King, Christopher T, and Pittman, Laura L. Performance Standards in Job Training and Job 
Placement Programs. Washington, DC: National Association of Counties, 1987, 35 pp. (ERIC 
Document Reproduction Service No. ED 3 10 259) 

Stern, David. Performance'Based Public Policy toward Postsecondary Vocational Education: 
Some Economic Issues. Berkeley: National Center for Research in Vocational Education, 
University of California, 1988, 41 pp. (ERIC Document Reproduction Service No. ED 315 
527) 

Hoachlander, E. Gareth; Choy, Susan R; and Brown, Cynthia L. Performance-Based Policies 
Options for Postsecondary Vocational Education: A Feasibility Study. Washington, DC: 
National Assessment of Vocational Education, U.S. Department of Education, 1989, 136 
pp. (ERIC Document Reproduction Service No. ED 3 15 535) 

Office of Technology Assessment. Performance Staruiards for Secondary School Vjcational 
Education: Background Paper. Washington, DC: Office of Technology Assessment, U.S. 
Congress, 1989, 80 pp. (ERIC Document Reproduction Service No. ED 313 591) 

Smith, Gregory R A Longitudinal Tracking Study of Short-Term Education and Employment 
Outcomes of Colorado Community College Graduates. Denver, CO: Community College 
and Occupational Education System, 1989. 



Florida Education and Training Placement Information Program. Annual Report. Tallahassee: 
Florida Department of Education, 1990. Figures 1-4, pp. 3-8, document the evolution of the 
pioneering multiagency state performance accountability program managed by Jay J. Pfeiffer. 
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Seppanen, Loretta. Vocational Outcomes in Washington Community Colleges: Baseline Re' 
port. Olympia: Washington State Board for Community College Education, 1990, 82 pp. 
(ERIC Document Reproduction Service No. ED 325 185) 
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Baj, John, and Trott, Charles E. A Feasibility Study of the Use of Unemployment Insurance 
Wage-Record Data as an Evaluation Tool for JTPA. Washington, DC: National Commis- 
sion for Employment Policy, 1991, 118 pp. (ERIC Document Reproduction Service No. ED 
329 749) 



Bross, Nancy. Issue Paper: Using Unemployment Insurance Wage Record Data for JTPA 
Postprogram Performance Standards. Chapel Hill, NC: Research and Evaluation Associ- 
ates, Inc., 1991, 21 pp. + appendices. 

Harmon, Tim. JTPA Performance Standards: Briefing for Illinois State Board of Education 
Committee of Practitioners. Springfield, IL: Job Training Programs Division, Department of 
Commerce and Community Affairs, 1991, 8 pp. 

Stevens, David W; Richmond, Peggy A.; Haenn, Joseph E; and Michie, Joan S. Measuring 
Employment Outcomes Using Unemployment Insurance Wage Records. Washington, DC: 
Office of Policy and Planning, U.S. Department of Education, 1992. (ERIC Document Re- 
production Service No. ED 354 345) 

Stevens, David W, and Shi, Jinping. New Perspectives on Documenting Employment and Earn' 
ings Outcomes in Vocational Education. Berkeley: National Center for Research on Voca- 
tional Education, University of California at Berkeley, 1996, 92 pp. (ERIC Document Repro- 
duction Service No. ED 399 398). Updates coverage of this literature through 1995. 

Sanchez, Jorge R., and Laanan, Frankie Santos, eds. Determining the Economic Benefits of 
Attending Community College, New Directions for Community Colleges, no. 104. San Fran- 
cisco, CA: Jossey-Bass Publishers, Winter 1998. Includes further updating of the postsec- 
ondary literature. 

Anderberg, Marc, and Pfeiffer, Jay J. A Field Guide to Automated FolloW'Up: Cost'Effective 
Collection of Performance Information. Austin: Texas State Occupational Information Co- 
ordinating Committee, 1998, 394 pp. 



The Legislative Sequence 



Ten vocational education and work force development laws are covered in chapters 2 and 3: 

• The Vocational Education Act of 1963 and the 1968 and 1976 Amendments 

• The Carl D. Perkins Vocational Education Act of 1984 and the Carl D. Perkins Voca- 
tional and Applied Technology Act Amendments of 1990 and 1998 

• The Comprehensive Employment and Training Act of 1973 and 1978 Amendments 
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The Job Training Partnership Act of 1982 
The Workforce Investment Act of 1998 



These laws led to three national studies of vocational education outcomes: 

1. The 1981 National Institute of Education final report 

2. The 1989 National Assessment of Vocational Education final report 

3. The 1994 National Assessment of Vocational Education final report 

The core indicator language in Perkins III and in the Workforce Investment Act Title I and 
Title II is covered in chapter 2. Chapter 3 examines the historical path that led to this common 
language. Chapter 4 explores what has happened on the federal and state accountability fronts 
in the 3 years since passage of the 1998 acts. 
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The Perkins III and 
WIA Title I and Title II 
Accountability Processes 



Contents of this chapter are as follows: 

• The federal-state partnership that is identifying exemplary state performance measure- 
ment approaches is described. 

• International Organization for Standardization and Baldrige National Quality Pro- 
gram quality management principles are used to identify federal and state opportunities 
to improve performance through adoption of refined measurement practices. 

• Some of the anticipated consequences of federal pursuit of common definitions across 
Perkins III and WIA Title I and Title II performance measures are explored. 

The content here reflects the author’s belief that release of core performance indicator data will 
increase public and media interest in many aspects of career and technical education and work 
force development system performance. There should be a sense of urgency at the federal, state 
and local levels to prepare to respond to the anticipated questions. 

A decade ago, Judith Gueron, President of the Manpower Demonstration Research Corpora- 
tion, described performance measurement criteria that are appropriate for the task that lies 
ahead (“Interview” 1991): 

Much of what evaluators can do to improve the use of their “product” in the 
policymaking process depends on how credible, clear, and useful evaluation informa- 
tion is. At bottom, we need to have a good product. Then we need to make our 
conclusions accessible and be able to say what they should (and should not) suggest 
to policymakers and practitioners. . ..At the early stages, we also need to be thinking 
about what policymakers and practitioners can use. (p. 76) 

Bers and Seybert (1999) identify seven major steps in the reporting process: 

1. Determining the clients, purpose, and audiences for the report 

2. Determining what data and information are available or must be gathered to satisfy the 
request 

3. Selecting the type of report 

4. Choosing the medium in which the report will be released 

5. Deciding how to depict data and information. “This decision is influenced not just by the 
client, purpose, audiences, type and format of the report, but also by the nature of the 
data and information” (p. 3). 
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6. Producing the report 

7. Disseminating the report 

This chapter concentrates on steps 1 and 2 from this list of seven steps for effective reporting. 
Step 2 should not begin until serious thought has been given to step 1. 

II Customers and Purposes for Performance Data 

Congress is an important client and audience for performance data. The Secretary of Educa- 
tion must submit an annual Perkins III report. The Secretary of Labor is required to submit an 
annual WIA report. These Secretaries, in turn, look to the Office of Vocational and Adult 
Education (OVAE) and the Employment and Training Administration (ETA), respectively, to 
provide the content of these reports. The federal offices are dependent upon states to provide 
the core indicator data and optional additional performance data needed to satisfy these con- 
tent requirements through parallel agencies. The state agencies rely upon separate local report- 
ing capacities. These local capacities reflect the ability and willingness of many students, teach- 
ers, administrators, and others to provide the necessary information. So, there are six layers of 
information handling from original data collection to final transmittal of findings to Congress. 
These are shown in figure 1. 

The arrows within each of the six layers of figure 1 indicate expected directions of flow for 
performance information release. The horizontal arrows represent release of performance infor- 
mation within the unit described and to outside constituents without further handling by oth- 
ers. A vertical arrow pointing up represents a hand-off of performance information to a higher- 
level authority. The higher-level entity then exercises control over whether, when, and how 
this performance information will be circulated internally and/or released to the public. A ver- 
tical arrow pointing down represents a return of performance information to those who had 
previously delivered data to the higher-level authority for processing and, perhaps, analysis. 

The gap between each pair of layers in figure 1 serves a descriptive purpose. Each gap represents 
an intermediate data processing step. 

• Original collectors of performance information at the base of the pyramid pass the data to a 
local consolidation agent — a school district, community college central office, or local 
Workforce Investment Board. 

• The local agents, and some private providers of occupational education and work force 
development training, direct this information through appropriate channels to a state 
agency — the State Department of Education, Higher Education Coordinating Board, or 
State Employment Security Agency, in most cases. 

• The state agencies combine information from multiple localities for transmittal to the proper 
federal agency — the Office of Vocational and Adult Education in the U.S. Department of 
Education and the Employment and Training Administration in the U.S. Department of 
Labor. 



24 




Processes 







t 

OFFICE OF VOCATIONAL AND ADULT 
EDUCATION/EMPLOYMENT AND 
TRAINING ADMINISTRATION 






Figure 1. Layers of performance Information handling from collection to transmittal to Congress 
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• Each of the federal units combines the state data that have been received into a report to 
their Secretary. 

• Each Secretary, in turn, submits the annual report to Congress. 

There are other audiences for performance data and other uses of performance information by 

authorities identified here. These additional customers and uses include — 

• The Cabinet members who retain some authority to allocate funds among programs within 
their domain 

• Program executives who manage pools of discretionary and incentive award funds 

• Mayors, council members, legislators, and governors who control state and local funds 
that can be used to motivate continual improvement of performance 

• Parents and students who must make future enrollment decisions 

• Business executives and human resources personnel seeking guidance about the quality 
of potential recruits from different schools and providers 

• Taxpayers seeking evidence of the return on their investments in career and technical 
education and work force training 

• Media reporters who often serve as a filter between a party that is releasing performance 
information and the various audiences identified 



The flow of performance information from collection to customer is more like a soaking hose, 
with many points of release, than like a pipeline with just one release of information to Con- 
gress. This difference is important. 

The content, timing, and format of release sought by customers are usually unique to each 
user’s intended purpose. Every step from basic data element definition through collection 
method(s) and quality control monitoring to processing specifications and presentation style 
should begin with explicit consideration of each customer’s requirements or preferences. 

Eleven types of performance indicator customer are identified here — Congress, Cabinet mem- 
bers, program executives, board members, school leaders, teachers, elected officials, parents 
and students, business executives and human resources professionals, taxpayers, and media 
people. Four core indicators of performance (Perkins III), 17 indicators (WIA Title I), or 3 
indicators (WIA Title II) cannot be expected to satisfy the needs of these diverse constituen- 
cies. 



One core indicator, “transition to further education or employment,” illustrates how different 
the needs of each type of customer might be. The illustration begins with a reexamination of 
the accountability language in Perkins III and WIA Title I and Title II. This is followed by an 
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exploration of what this means for preferred data element definitions and collection methods, 
and recommended quality control procedures. 

Transition to Turther tducation and/or tmpioyment 

Perkins 111, Section 113(b)(2)(A), defines Core Indicator Three as “placement in, retention in, 
and completion of, postsecondary education or advanced training, placement in military ser- 
vice, or placement or retention in employment.” 

WlA Title I, Section 136(b) (2) (A) , defines Core Indicator Nine as “older youth (eligible youth 
aged 19-21 at registration) entered employment rate.” 

WlA Title 11, Section 212(b) (2) (A) , defines Core Indicator Tioo as “placement in, retention in, 
or completion of, postsecondary education, training, unsubsidized employment, or career ad- 
vancement.” 

Review these three core indicator definitions. Decide how many possible transition events ap- 
pear in the three definitions. A “transition event” is a change from one status to another, such 
as moving from school to work or from high school to a community college. What number of 
transition events did you choose? 1 selected the number 9, with an asterisk. 

The Perkins 111 Core Indicator Three definition appears to include four clear transition event 
possibilities, which are not mutually exclusive: 

1. Placement in postsecondary education 

2. Placement in advanced training 

3. Placement in the military 

4. Placement in employment 

The WlA Title 1 Core Indicator Nine definition includes one transition possibility: entered em- 
ployment. 

The WlA Title 11 Core Indicator Two definition appears to include four transition event possi- 
bilities: 

1. Placement in postsecondary education 

2. Placement in training 

3. Placement in unsubsidized employment 

4. Career advancement 

Compietion 



The asterisk was added to indicate uncertainty about whether “completion” should be treated 
as a transition event. “Completion” of a career and technical education program is assumed to 
increase a student’s value in the world of work. But the realization of this value requires another 
step, actual transition to employment. 
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Debate continues about whether educational accountability should properly end with evidence 
of skill development, without consideration of the later use of these skills. Skill use is contin- 
gent upon factors over which educators exercise little influence. 



“Completion” per se is a solid core indicator of performance for some customers of an account- 
ability system, such as Congress, the Office of Vocational and Adult Education, a state depart- 
ment of education, a principal, or a teacher. But legislators, parents and students, business 
human resource professionals, school board members, and journalists are expected to look be- 
yond this intermediate stage of achievement. These customers are likely to focus on the transi- 
tion to work or further education of “completers.” 

The word “completion” in Perkins III Core Indictor Three led to immediate confusion for post- 
secondary performance measurement. There is apparent overlap with Core Indicator Two. The 
definition of Core Indicator Two is — 



student attainment of a secondary school diploma or its recognized equivalent, a 
proficiency credential in conjunction with a secondary school diploma, or a postsec- 
ondary school degree or credential. 

The Office of Vocational and Adult Education resolved this problem by not including comple- 
tion as a subindicator under Core Indicator Three in the Core Indicator Framework document 
(U.S. Department of Education 2000). “Postsecondary degree or credential” remains a 
subindicator under Core Indicator Two. 



Retention 




The word “retention” appears in each of the three core indicators defined here — Perkins III 
Core Indicator Three, WLA Title I Core Indicator Nine, and WIA Title II Core Indicator Two. 
Retention is not a transition event. There has been no movement from a beginning status to a 
new destination status. 

Retention can be an important indirect indicator of progress toward a goal. For example, the 
Perkins III Core Indicator Three phrasing is “placement in, retention in, and completion of, 
postsecondary education or advanced training.” Retention is a necessary intermediate step be- 
tween placement and completion. 

Does the distinction between continuity in a job held before and after completion of a career 
and technical education program, versus documented entry into a new job after completion, 
matter? It is important to know the intended use of the performance information before an- 
swering this question. 

Congress may not care about the distinction. The stated purpose of Perkins III, in Section 2 of 
the act, is — 

to develop more fully the academic, vocational, and technical skills of secondary and 
postsecondary students who elect to enroll in vocational and technical education 
programs. 
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There is no explicit reference to an expected employment outcome, but it is reasonable to 
assume that development of academic, vocational, and technical skills is an intermediate step 
toward use of these skills in employment. 

The stated goal for state performance accountability in Section 1 13(a) of the act is — 

to assess the effectiveness of the State in achieving statewide progress in vocational 
and technical education, and to optimize the return of investment of Federal funds 
in vocational and technical education activities. 

Again, this language can be interpreted to mean that educational accountability should be 
limited to indicators of a state’s progress in developing academic, vocational, and technical 
skills of only those students who elect to enroll in vocational and technical education programs. 
Consistent with this interpretation, optimization of federal funds allocations would be limited 
to indicators of relative progress among the states in developing these skills for the designated 
students. 

The phrase “placement or retention in employment” (emphasis added), as a core indicator 
component in Section 1 1 3 (b) (2) (A) of the act, reinforces a conclusion that Congress was not 
interested in the distinction in October 1998, when Perkins III became law. 

Commas and the Words “And” and “Or” 

A difference of phrasing appears in the Perkins III and WIA Title II core indicators. The Perkins 
III Core Indicator Three definition is “placement in, retention in, and completion of, postsecond- 
ary education or advanced training” (emphasis added) . 

The WIA Title II Core Indicator Tivo definition is “placement in, retention in, or completion of, 
postsecondary education, training” (emphasis added). 

The difference in phrasing can be interpreted as a signal of the intent of the Committee staff 
niembers who crafted the final wording of Perkins III and WIA Title II. 

• Vocational and technical education students are expected to progress from placement, 
through retention, to completion of postsecondary education or advanced training. 

• Adult education and literacy program enrollees are expected to advance only one step, 
depending on their previous postsecondary status. 

“Placement” versus “entered employment” 

The words “placement,” “retention,” and “completion” appear in Perkins III Core Indicator 
Three and in WIA Title II Core Indicator Two, but not in WIA Title I Core Indicator Nine, where 
the term “entered employment” appears instead. Do “placement” and “entered employment” 
mean the same thing? We know that today the answer is “no”; these are not synonyms in 
practice. 

O 
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The Perkins III Core Indicator Three does not require a student to make the transition into 
employment after completion of a secondary or postsecondary career and technical education 
program to satisfy the “placement or retention in employment” criterion. Reporting evidence 
that a former student is employed within an agreed-upon time interval after program comple- 
tion satisfies the core indicator requirement for federal Perkins III accountability. 



After leaving high school, a former student can be found in the same part-time job s/he has held 
continuously since entering a career and technical education program, or before, and be counted 
as a “placement or retention in employment” success. In fact, a former high school career and 
technical education program completer who enrolls part time in a local community college and 
works part time might be counted as a “placement or retention in employment” and as a “place- 
ment in postsecondary education,” if quality control steps to avoid this are not taken. One goal 
of the OVAE Data Quality Initiative, which is described in more detail later in this chapter, is 
to encourage states to recognize the importance of nonduplication of this kind for federal re- 
porting purposes. 

Concurrent employment and Postsecondary enrollment 

An annual cycle of postsecondary education data collection is common. Assume that a compl- 
eter of a high school career and technical education program graduates at the typical time of 
year. S/he then enrolls at a local community college in the Fall, continues in the Winter/Spring 
term, and works part time throughout the year. S/he would be expected to appear in records 
that are often used to document these three subindicator measures for Core Indicator Three 
calculations. 

The feasibility of identifying and eliminating duplication depends upon the methods and tim- 
ing of data collection that are used and adoption of quality control steps to eliminate the possi- 
bility of counting a former career and technical education student in more than one perfor- 
mance indicator. A basic requirement is that each source of performance information must use 
a common identifier, so the sources can be linked to detect multiple appearances. 

Choosing among Transition Destinations 



No rule has been established by OVAE to advise states how to proceed when multiple appear- 
ances occur in performance data files. There has been no formal expression of a federal prefer- 
ence among transition possibilities. 



Some states have defined a high school graduate’s enrollment in postsecondary education as 
preferred over immediate transition to employment. In such performance accountability situa- 
tions, the large subpopulation of former high school vocational and technical education 
completers found in both postsecondary education and employment would be assigned to post- 
secondary education only. The documented employment status would be ignored for this per- 
formance accountability use of the data. 



A public ranking of preferred student transitions offers users of performance information a 
clear understanding of the short-term goals set for former career and technical education stu- 
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dents. The Perkins III Core Indicator Three composite does not convey this important informa- 
tion about educational policy objectives. Again, the definition of Core Indicator Three is — 

placement in, retention in, and completion of, postsecondary education or advanced 
training, placement in military service, or placement or retention in employment. 

Literal versus Expedient Interpretations 
of the Accountability Language 

Linguistic care is not always exercised at the time of final crafting of forward-looking perfor- 
mance accountability language. Members of pertinent committees, and staff members respon- 
sible for the negotiation of phrasing, understandably concentrate ori funds allocation and stu- 
dent eligibility criteria at this occasionally frantic point in negotiation of compromise language. 

Members of Congress have no obligation to limit questions during reauthorization hearings to 
the indicators that were placed in an act some time in the past. Symmetry is not required 
between forward-looking goal statements and backward-looking questions about performance 
in the pursuit of these goals. 

Return to the six layers of performance information handling in figure 1 (p. 25) and to the 10 
bullets listing types of performance information customer that follow figure 1. Is a “placement 
or retention in employment” indicator expected to satisfy any of these customers, including 
members of Congress? My answer is yes and no. Yes, it satisfies a minimum core indicator 
requirement. But, no, this is not expected to satisfy anyone’s curiosity about the postcompletion 
employment of former students in career and technical education programs. 

The importance of “Comparison ” Requirements 

One word captures the essence of why confidence is expressed here about the inadequacy of 
Core Indicator Three. The word is “comparisons.” Perkins III, Section 113(c)(3), states that the 
Secretary of Education — 

(A) shall make the information contained in [State annual] reports available to the 
general public; (B) shall disseminate State -by-State comparisons of the information; 
and (C) shall provide the appropriate committees of Congress with copies of such 
reports. 

This sharing of performance information is expected to have at least three important effects on 
state performance accountability systems: 

1. States have an incentive to include only core indicator information in the required annual 
report to the U.S. Department of Education. This limits each state’s exposure to possibly 
unfavorable comparisons with the unknown performance of other states. 



2. States may also have an incentive to include optional additional indicators of performance. 
However, if discretionary indicators are included in a state’s approved plan, measures of 
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performance using these indicators must be included in the state’s annual performance re- 
port to the U.S. Department of Education. 

States are most likely to accept this offer if favorable performance is expected. But each 
state has better information about its own expected performance than how other states will 
perform. States that are satisfied enough with their performance to voluntarily place the 
evidence of this performance in national public circulation are more likely than less-confi- 
dent states to include optional indicators of performance in their plan and subsequent an- 
nual performance reports. 

Definitions of optional indicators are not included in the act, because Section 1 13 (b) (3) (A) (i) 
is designed to encourage state creativity in the selection of a broad portfolio of performance 
indicators. Each state retains exclusive control over the choice of additional indicators and 
the definition of each of these, subject to OVAE acceptance in negotiations. 

3. States that have chosen not to provide performance information beyond the minimum re- 
quired core indicators will be asked about this cautious decision. Less -forthcoming states 
will be told about the more expansive portfolios of performance indicators submitted by 
some other states. 

Who might ask these questions? Anyone who becomes aware of uneven availability of perfor- 
mance information among the states, including Congressional committee staffs, the General 
Accounting Office, state legislators, and particularly media representatives who seek access to 
more and better information. 



What can be done to prepare for the questions that will be asked? The remainder of this chap- 
ter concentrates on answering this question. 

1 1 The OVAE Data Quality Initiative 



The Program Reform Branch in the Office of Vocational and Adult Education (OVAE) of the 
U.S. Department of Education knew that some states had begun to cooperate in voluntary 
movement toward a common alignment of basic performance indicator definitions and prac- 
tices long before final passage of the 1998 Perkins III Amendments. Soon after the amend- 
ments were signed into law in October 1998, OVAE began a formal Data Quality Initiative. 

The first step in the Data Quality Initiative was to separate the 4 core indicators of perfor- 
mance into 14 subindicators. This was done by OVAE staff in consultation with the states and 
other interest groups. The 14 subindicators are as follows: 



Core Indicator 1 : Skill Proficiency Attainment 



LSI 

1.S2 

l.Pl 

1.P2 



Secondary academic attainment 

Secondary vocational and technical skill attainment 

Postsecondary academic attainment 

Postsecondary vocational and technical skill attainment 
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Core Indicator 2: Credential Attainment 

2.51 Secondary completion 

2.52 Secondary proficiency credential with secondary diploma 

2. P1 Postsecondary degree or credential 

Core Indicator 3: Placement and Retention 

3 .S 1 Secondary placement 

3. PI Postsecondary placement 

3. P2 Postsecondary retention 

Core Indicator 4: Participation in and Completion of NontracUtional Programs 

4.51 Participation in secondary nontraditional programs 

4.52 Completion of secondary nontraditional programs 

4. P1 Participation in postsecondary nontraditional programs 
4.P2 Completion of postsecondary nontraditional programs 

The interdependence of Management Information 
System Components 

The first sentence of chapter 1 of this volume refers to a “new generation of the performance 
accountability partnership among the states, the U.S. Department of Education and the US. 
Department of Labor.” The list of 14 subindicators just presented highlights the importance of 
another new generation accountability partnership — -between secondary and postsecondary 
management information system personnel in each state. The success of the OVAE Data Qual- 
ity Initiative will depend, in many ways, upon the ability of these and other data management 
systems to talk to each other. 

Figure 1 uses a single data flow pyramid composed of six layers of data transfer between collec- 
tion and reporting to Congress. Separate data flows pertaining to vocational and technical 
education and employment training are represented in this single pyramid. Now, with the in- 
troduction of the 14 subindicators of vocational and technical education performance, it helps 
to think of many pyramids of data collection and transfer prior to arrival in the hands of a 
customer. 



At a minimum, there are public secondary, public postsecondary, and work force development 
data collection, processing, and release pyramids (alternatively referred to as “silos” to capture 
their free-standing independent nature). There are multiple hidden passageways of data collec- 
tion and internal transfer within each of these data flow systems. 



Data collection under the Elementary and Secondary Education Act will be revised when the 
No Child Left Behind Act of 2001 emerges from conference, is passed and signed into law. The 
Improving America’s Schools Act of 1 994 and the Individuals with Disabilities Education Act of 1 997 
amendments had already provided a context of management information system change before 
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passage of the Perkins III Amendments in 1998 (Goertz, Duffy, and Le Floch 2001). The Stu- 
dent Right to Know and Campus Security Act of 1991 requires institutions eligible for Title IV 
federal financial aid to release graduation rates of their first-time, full-time students. The Ntt- 
tional Education Statistics Act of 1 994 requires the National Center for Education Statistics (NCES) 
of the U.S. Department of Education to “collect, analyze, and disseminate statistics and other 
information related to education.” 

The reporting requirements that flow from each of the acts mentioned, and others, can be 
thought of as modules of a larger information management strategy. These modules are often 
independent, allowing change without affecting other modules. At the same time, particular 
components have not necessarily been designed to permit routine linkage with other modules. 

The OV/it Phot Test of Statewide Performance indicator 
Reporting Capacities 

Eight states (Florida, Illinois, Indiana, Missouri, New Jersey, Ohio, Texas, and Virginia) volun- 
teered to test their capacity to report statewide performance for secondary academic attain- 
ment, vocational and technical skill attainment, award of a secondary diploma, and placement 
in postsecondary education, military service, or employment. 

The pilot phase goals were to^ 

• Identify states’ capacities to provide data for each of the core indicators of performance 

• Determine differences in state definitions of threshold levels and performance measures 

• Evaluate the sensitivity of performance results with respect to states’ reporting approaches 

Secondary Piiot Phase Tindings and Remaining issues 

In the Secondary Pilot Project on Accountability, Vickie Schray (1999) includes the following 
findings: 

There are significant differences in state definitions of threshold levels — concentra- 
tors — as well as state definitions of completers. There is a need to further clarify state 
definitions for more accurate and reliable state comparisons, (p. 3) 

The most significant differences between states are in their use of concentrators or 
completers in their denominators for the skill attainment and completion measures. 

These differences may have significant effects on state performance differences, (p. 

3) 

Schray identified major remaining issues: 

• Improving clarity and consistency in state definitions — to identify common definitions where 
possible and better understand and document state differences when reporting performance 
data. 
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• Defining the denominator in performance measures — to reach greater consensus in defin- 
ing the student populations to be included in the denominators of the core measures. Con- 
centrators or completers for academic attainment? All completers or only completers who 
graduate for placement? 

• Reporting state differences and improvements in state data quality — especially in areas that 
affect state performance comparisons (e.g., skill attainment, placement) within and between 
measurement approaches. 

Postsecondary Pilot Phase rindings and Remaining issues 

In Postsecondary Pilot Project on Accountability, Schray (2000) includes the following find- 
ings: 

There are significant differences in state definitions of threshold levels — -vocational 
concentrators/program enrollees. (p. 2) 

These differences have significant effects on the percentage of vocational partici- 
pants who are reported as vocational concentrators/program enrollees. (p. 2) 

The most significant differences in state definitions are — 

“Entry vs. Exit Cohort” — States may define the cohort based on when they en- 
rolled in a program and set a time to degree time period (e.g., 5 years) for report- 
ing completion rates of that entry cohort [or] States may define an exit cohort 
based on when students stopped participating in a program, (p. 2) 

“First-time, Full-time” — States may include all program enrollees, only full-time 
enrollees, or only first-time, full-time enrollees. (p. 3) 

“Enrollment Definition” — States may vary in how they define program enrollees 
(i.e., vocational concentrators). States may require student intent to complete a 
program and may require a minimum number of credits earned ranging from one 
course to three-fourths the credits needed to complete a program, (p. 3) 

“Tracking Time Period”— States may vary in the time period they allow students 
to complete a program before they are counted as a noncompleter, (p. 3) 

Schray identified major remaining issues: 

• Improving clarity and consistency in state definitions 

• Measuring program completion 

• Measuring placement 

• Reporting state differences and improvements in data quality 
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Steps Taken after the Pitot Phase Tindings Were Known 

The results of the pilot test of state performance reporting capacities influenced the content of 
the OVAE Core Indicator Framework, which was released in January 2000. The intended uses 
of the framework were clearly stated in the introductory paragraphs of the paper; 

OVAE will use this framework to provide clear guidance and technical assistance to 
states. The framework will also be the basis for performance reporting required by 
Perkins 111.... OVAE views the development of this core indicator framework as the 
first step in working with states, the Department of Labor, and stakeholder organiza- 
tions to develop a larger performance accountability framework. This larger frame- 
work will address the remaining performance accountability system issues including 
setting and adjusting performance levels, continuous improvement, and incentive 
grants and sanctions. (U.S. Department of Education 2000, p. 1) 

A series of technical assistance workshops for the Perkins 111 state performance accountability 
teams, convened by OVAE, followed the release of the Core Indicator Framework. A Web- 
based Peer Collaborative Network was established by OVAE as a basic activity of the Data 
Quality Initiative. This continues to offer rapid-response answers to questions posed by state 
performance accountability team members, including counsel from state peers as well as federal 
overseers of the Perkins 111 performance accountability process. A train-the-trainer technical 
assistance workshop for the state performance accountability teams was sponsored by OVAE in 
late August 2001. This broadened and deepened the pool of federal and state accountability 
expertise. The flow of learning is both to and from the states in these workshops. 

The OVAE Data Quality Initiative has not occurred in isolation. One reason is that the Gov- 
ernment Performance Resides Act of 1 993 (GPRA) required all federal agencies to establish per- 
formance accountability systems. Each agency is expected to report on a regular basis to Con- 
gress on progress achieved in pursuing performance goals that have been set. The Baldrige 
National Quality Program and International Organization for Standardization principles have 
been adopted by many agencies in responding to the GPRA requirements. 

This broad context for federal program performance review is reflected in the next two sec- 
tions. The first section covers highlights from a recent General Accounting Office (GAO) 
report to Congress on Department of Labor GPRA accomplishments. The second section con- 
tains some basic principles of outcome -based performance measurement that were prepared by 
an outside contractor for the Department of Health and Human Services. These presentations 
are intended to establish a solid foundation for application of the Baldrige and ISO manage- 
ment quality principles. 

1 1 Selected GAO Tindings from 
a Review of DOL Performance 



The GAO report to Congress (2001), titled DEPARTMENT OF LABOR: Status of Achieve 
ing Key Outcomes and Addressing Major Management Challenges, describes how the assess- 
ment was conducted: 
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As agreed, using the selected key outcomes for Labor as a framework, we (1) assessed 
the progress Labor has made in achieving these outcomes and the strategies the 
agency has in place to achieve them; and (2) compared Labor’s fiscal year 2000 
performance report and fiscal year 2002 performance plan with the agency’s prior 
year performance report and plan for these outcomes, (p. 3) 

With respect to the planned outcome “job training participants get and keep jobs,” the GAO 
concluded that — 



Labor reported making progress in achieving this outcome, with performance meet- 
ing most of the goals. However, assessing progress for some goals has been compli- 
cated by the transition from one employment and training program to another and 
by the lag in available performance data. (p. 4) 

Another of the four key outcomes selected by GAO for review is “individuals successfully tran- 
sition from welfare dependency to self-sufficiency.” GAO’s assessment of Labor’s performance 
is instructive here, because it addresses quality criteria in the chosen performance indicator 
definition and a quality criterion for the measurement process itself 

Labor improved the goal from last year’s report by incorporating a measure for job 
retention and one for increased earnings. The strategies identified to achieve the 

goal appear to be clear and reasonable and incorporate program evaluation Labor 

improved its fiscal year 2000 performance report from its fiscal year 1999 report by 
providing more complete information on the quality and credibility of performance 
data.... Labor used a new assessment category — “substantially achieved” — when at 
least 80% or more of the goal was attained.... The use of this category may give an 
impression of successful performance across all goals or on a specific goal when ac- 
tual performance might be as much as 20 percent below the target... .Labor’s fiscal 
year 2002 plan improved its discussion of its efforts to verify and validate perfor- 
mance data as compared with the fiscal year 2001 plan. However, it still lacks suffi- 
cient information on how it will address certain data limitations, such as timeliness 
and coverage limitations, (p. 5) 

The “retention” and “earnings gain” subindicators of performance described in the previous 
paragraph are relevant to the “transition from welfare to self-sufficiency goal.” Self-sufficiency 
is a more rigorous target than employment. 



Close attention should be given to these GAO conclusions about the quality of Department of 
Labor performance measurement. The same issues are expected to appear in reviews of other 
federal programs. 



The Department of Health and Human Services (DHHS) sought the assistance of an outside 
contractor to outline basic principles of outcome-based performance. Alternative Outcome 
Measures: TANF Block Grant, Chapter 2: Principles of Outcome-Based Performance Mea- 
surement (U.S. Department of Health and Human Services 2000) can be found at http:// 
www.aspe.hhs.gOv/hsp/alt-outcomes00/Ch2.htm. The contractor’s findings are briefly presented 
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Principles of Outcome-based 
Performance Measurement 



An outcome 'based performance measurement system consists of four basic components: goals, 
measures, standards, and consequences. 

A Goa! Statement 

A clear goal statement is required at the outset, so there can be a common understanding of the 
outcome that is sought. Without consensus about a goal, it is impossible to define measures of 
performance that can be used to gauge progress toward the goal. The criterion of common 
understanding does not require universal buy-in, only that there is a common understanding of 
the goal. 

Validity and Reliability of Measurement 

A good indicator of performance should have two basic attributes: 

1. Validity — general agreement that the indicator is, in fact, an acceptable measure of per- 
formance with respect to a previously stated goal 

2. Reliability — consensus that the chosen indicator offers an accurate measurement of per- 
formance, such that repeated use of the indicator under identical conditions would pro- 
duce the same result, and that adoption of the measure under different circumstances 
will produce an accurate and valid measurement of performance 

Validity and reliability are separate criteria in judging the appropriateness of a particular perfor- 
mance indicator. For example, there could be a consensus that Unemployment Insurance Wage 
Records offer an accurate measurement of placement in employment (a reliability criterion), 
while at the same time there could be persistent disagreement about this being an acceptable 
measure of performance for an act with a stated purpose “to develop more fully the academic, 
vocational, and technical skills of secondary students and postsecondary students who elect to 
enroll in vocational and technical education programs” (a validity criterion). In fact, a consen- 
sus has not emerged about the reliability of Unemployment Insurance Wage Records for this 
performance measurement purpose. Some people accept “placement in employment” as a valid 
indicator, but question the reliability of this particular way of measuring performance on this 
subindicator. Others are more comfortable with the reliability of this administrative data source 
than with employment as a valid measure of vocational and technical education performance. 

These disagreements will surface in upcoming reauthorization negotiations. This is one reason 
why close attention has been urged by the OVAE staff and other federal and state partners in 
accountability reporting to the validity and reliability principles of performance measurement 



quality. 
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Standards, or Levels, of Performance Targeting 

The “standards” or “levels” component of an outcome-based performance measurement sys- 
tem should contain only valid and reliable indicators. These can then be used to define interim 
target values to gauge progress toward a previously stated goal. 

AfUusted Peiformance Targets 

The adoption of measurement standards, or levels, to reflect differences in conditions that are 
agreed to have an impact on performance, increases the difficulty of “standards" or “levels” 
specification. Adjustments are needed for equity in performance measurement but there is little 
agreement about what the adjustment factors should be and how they should be measured. 

The simplicity of core indicators that are intended for release to the public is compromised by 
the introduction of adjustments. This is a serious issue for those who define the measures, those 
who carry out this mandate, and those who use the resulting performance information. At what 
cost has simplicity been achieved? Who are the winners and losers from too little recognition of 
adjustments of performance targets? Partial answers are given to these questions later in this 
chapter and in the concluding chapter. 

The Consequences of Measured Performance 

Neglect of the consequences component of an outcome -based performance measurement sys- 
tem is a direct route to peril. Those who have invested in consensus building about a goal, 
specification of valid and reliable measures of progress toward this goal, and negotiation of 
interim target levels of these measures, should expect different consequences to follow success 
or failure in reaching or exceeding this target level during a reporting cycle. 

If a major element in the payoff on the investment made in performance measurement quality 
is not consequences, then the outcome -based performance measurement system is severed from 
subsequent policy and management decisions. Why invest in quality of performance measure- 
ment if performance itself does not matter? 

A Process Criterion for Performance Measurement Quality 

The DHHS contractor identified a different kind of performance measurement quality crite- 
rion: 



Engaging a broad range of stakeholders in structured discussions about all of these 
elements [goals, measures, standards and consequences] can be the basis for build- 
ing performance partnerships in designing and implementing a performance mea- 
surement process. Even if one is not trying to achieve consensus, this wider audience 
provides a better perspective on what is possible in the real world (e.g., what is opera- 
tionally feasible, what are possible unintended consequences, etc.) and may facilitate 
future data collection efforts. (U.S. Department of Health and Human Services 2000, 
p.l) 
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The contractor drew relevant lessons from the research conducted for DHHS: 

By necessity, performance measurement systems are limited to those elements for 
which data can be collected inexpensively, routinely, and in a timely fashion, (p. 4) 

Outcome-based performance measurement can still be a useful tool to monitor pro- 
gram operations and promote improvements, as long as stakeholders at all levels of 
operations agree that there is a clear logical system connecting the activities of pro- 
gram operators to the outcomes that are measured, (p. 4) 

The chosen measures must not give programs incentives to achieve high levels on 
performance measures through the use of strategies that subvert their fundamental 
intent, (p. 4) 

Because it is impossible to fully account and adjust for all the variations in circum- 
stances among states, no performance measurement system can be perfectly fair. It is 
important to develop mechanisms which recognize that states are facing different 
economic and demographic environments, (p. 4) 

It is extremely difficult to determine an appropriate standard without baseline data 
on past performance. When data for a specific measure have never been collected or 
analyzed before, neither state nor federal policymakers are likely to know what would 
be a reasonable level of performance, (p. 5) 

An additional factor must be considered when a new performance measurement 
system is adopted. . . .When data for a new measure are first collected, in many cases, 
states will have little ability to predict their performance in advance — either because 
the program is new and there is no past performance, or because the data collection 
requirement is new and there are no baseline data. This uncertainty about perfor- 
mance levels appears to have very different consequences depending on whether a 
bonus or penalty is involved. In the context of penalties, performance uncertainty 
appears to lead to highly risk-averse behavior.... In the context of bonuses, uncer- 
tainty appears to lead to a wait-and-see attitude. . . .One possible means of mitigating 
the negative consequences of this asymmetry would be to implement a new mea- 
surement system in phases, beginning first with bonuses for high performers and 
adding penalties only after several years of experience with the measures, when more 
information would be available to use in setting standards, (p. 8) 

The performance measurement lessons highlighted here are expressed in the DHHS contractor’s 
words to emphasize a point. The Perkins III performance accountability process undertaken by 
OVAE through the Data Quality Initiative has followed basic “good practice” principles of 
system design that apply far beyond vocational and technical education activities. 
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Applying Quality Management Principles to the 
OVAE Data Quality Initiative 

The Baldrige Education Criteria for Performance Excellence (U.S. Department of Commerce 
2001) and International Organization for Standardization (ISO) Principles for quality manage- 
ment assessment exhibit substantial overlap, as would be expected. No attempt is made here to 
apply these criteria and principles sequentially. 

The Baldrige criterion “information and analysis” and the ISO principle “a factual approach to 
decision making” both focus on data accuracy, accessibility, and proper use. The ISO principle 
adds “taking action based on factual analysis, balanced with experience and intuition,” whereas 
the Baldrige criterion adds “evidence of their use and effectiveness in daily operations” and 
“steps taken to keep the performance measurement system current with service needs and 
directions.” 

Data Accuracy, Accessibility, and Use 

Perkins III, Section 3, Definitions, does not include a definition of “vocational and technical 
education students,” even though this term appears in the Section 2 statement of the act’s 
purpose. A goal cannot be translated into an action plan without a clear understanding of the 
target population for such actions. 

Section 3 (29) does define “vocational and technical education” as — 

organized educational activities that (A) offer a sequence of courses that provides 
individuals with the academic and technical knowledge and skills the individuals 
need to prepare for further education and for careers (other than careers requiring a 
baccalaureate, master’s, or doctoral degree) in current or emerging employment sec- 
tors; and (B) include competency-based applied learning that contributes to the aca- 
demic knowledge, higher-order reasoning and problem-solving skills, work attitudes, 
general employability skills, technical skills, and occupation-specific skills, of an in- 
dividual. 

Early in the development of the Core Indicator Framework, OVAE defined four sequentially 
more demanding stages of secondary vocational and technical education attainment: 

1. Participant — a student who enrolls in at least one vocational and technical education 
course 

2. Concentrator — a “participant” who reaches a state-defined threshold of vocational and 
technical education courses 

3. Completer — a concentrator who acquires a state -defined portfolio of academic and tech- 
nical knowledge, skills, and proficiencies 
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4. Graduate — a completer who also receives a high school diploma or its state-recognized 
equivalent 

Each of these definitions determines which subpopulations of students, or former students, are 
to be included in the denominator of each of the secondary subindicators of performance found 
in the Core Indicator Framework. 

The Definition and Measurement of “Concentrator” 

State information systems define vocational and technical education programs. The new term 
“concentrator” did not always align with existing state information systems. The new term 
often fell between state-defined courses and clusters of courses called programs, and was there- 
fore not defined for management reporting purposes. 

The performance accountability goal is clear — ^winnow the population of all vocational and 
technical education participants to remove the casual isolated course-taker from the pool of 
those who are making a serious attempt to acquire technical knowledge and skills needed for 
further education and employment success. 

From the perspective of the Secretary of Education, Congress, and the public, it is important for 
OVAE to ensure that each state defines concentrator to include all students who satisfy the 
criterion “making a serious attempt.” This definition is the practical starting point for Perkins 
III performance accountability. 

The Definition and Measurement of “Compieter” 

Notice an important difference in the OVAE criteria that define “concentrator” and “compl- 
eter.” The concentrator definition is based on a student’s reaching a course-taking plateau. The 
completer definition is based on attainment of a state-defined portfolio of academic and tech- 
nical knowledge, skills and proficiencies. 

One of the difficult challenges for OVAE in its encouragement of state convergence toward 
common definitions of the core indicators is that most state accountability systems today define 
both concentrator and completer based on attainment of specified vocational and technical 
education course completion points. But the availability of a reliable data link between a state- 
defined course completion point and student attainment of academic and technical knowl- 
edge, skills, and proficiencies differs among the states. This unevenness is a key target of the 
OVAE Data Quality Initiative. 

The Office of Technology Assessment’s (1994) Testing and Assessment in Vocational Educa- 
tion is an excellent source for a pre-Perkins III treatment of this issue. The National Skill 
Standards Board website (www.nssb.org) is a good source for up-to-date coverage of progress 
that has been made in defining common national standards for some industries and occupa- 
tions. 
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A common, and contentious, issue with regard to the definition of secondary graduation is the 
definition of a “state -recognized equivalent” of high school graduation. Some states treat satis- 
factory performance on the Tests of General Educational Development (GED Tests) as the 
equivalent of a high school diploma. The GED Tests, developed by the American Council on 
Education, measure attainment in writing skills, social studies, science, interpretation of litera- 
ture and the arts, and mathematics. The specification of scoring criteria to be awarded a passing 
score overall is left to each state. 

Those who pass the GED tests in some states are awarded a regular high school diploma. So, a 
state’s performance information system may not distinguish between those who passed the 
GED tests and those who completed all regular, or routine, high school graduation require- 
ments. 

Does such a definition of “graduate” satisfy the validity and reliability criteria defined earlier for 
a “good” performance indicator? The GED tests route to high school graduation may be grow- 
ing in importance. Until a recent cooling of economic performance in many local economies 
across the United States, anecdotal evidence had surfaced in “hot” local labor market situa- 
tions that some high-achieving students were learning marketable skills, then dropping out of 
school to accept an attractive job offer, and later passing the GED tests to satisfy a postsecond- 
ary enrollment requirement. This phenomenon appears as leakage from the data collection and 
transfer pipeline through which traditional progress from concentrator status to completer, and 
then on to graduate, is documented. 

Only former vocational and technical education concentrators can complete. So, if only 
completers are counted in the denominator of a placement rate indicator, premature placement 
prior to completion does not appear in a traditional vocational and technical education man- 
agement information system. 

Up to this point, only student subpopulation definitions that surfaced during the OVAE pilot 
test of Perkins III secondary core indicators have been discussed — ^vocational and technical 
education concentrators, completers, and graduates. These are the sequentially more demand- 
ing student attainment criteria that define the denominator building blocks for calculating 
Perkiris III secondary subindicator values. 

Having covered the “who” of Perkins III accountability, attention turns now to the “what.” The 
limited information sought from states during the OVAE pilot phase included secondary aca- 
demic and vocational-technical skill attainment, secondary credential attainment, and place- 
ment. 
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The Definition and Measurement of 
“Secondary Academic Skiii Attainment” 



The explicit performance measurement goal is to document student mastery of academic knowl- 
edge. Two state-level decisions are involved: 

1. Define the concentrator subpopulatiori that will be used in reporting secondary academic 
skill attainment. 

2. Define the challenging state academic standard that determines which concentrators 
counted in the denominator are included in the numerator as having met this standard. 







The practical objective in each annual cycle of subindicator reporting is to calculate a percent- 
age figure. What percent of the reference subpopulation of “concentrators” met an interim 
plateau of academic knowledge? 

The goal defined in the 1998 amendments is a level of knowledge attainment. This differs from 
the performance goal sought under the 1990 act, which focused on performance gciin. 

One perspective on the timing of academic knowledge mastery, as a Perkins III core subindicator, 
is that this should be measured before a student enrolls in intermediate and upper-level voca- 
tional and technical education courses. This point of view holds that a threshold level of aca- 
demic attainment is a necessary prerequisite to success in vocational and technical skill attain- 
ment. 



A different point of view holds that academic knowledge attainment should be measured coin- 
cident with or after a student’s exit from vocational and technical education studies. Student 
attainment of academic and vocational-technical skills is seen as the result of a single inte- 
grated process that cannot be separated into before and after components. The relevance of 
particular course prerequisites is not denied, but attention is directed toward a student’s attain- 
ment at the time they would be expected to transition. 

The cumulative effect of measurement challenges, including the ones that have been intro- 
duced here, might seem to threaten the integrity of the federal-state performance indicator 
partnership. State accountability systems measure academic attainment in different ways. State 
information systems undergo frequent modifications on different time schedules in response to 
mounting pressure to compare and contrast student achievement levels. Anticipated passage 
of the No Child Left BeMnd Act of 2001 will increase the frequency of information system modi- 
fication. 

The Definition and Measurement of “Secondary 
Vocationai and Technicai Skiii Attainment” 



The goal is to document student mastery of vocational and technical skills. OVAE has defined 
this to mean “knowledge and skills that meet State- established and industry-validated stan- 
dards.” 
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A frequent state response to the OVAE definition is that current state definitions of completer 
and graduate satisfy this performance indicator requirement — students who complete, and par- 
ticularly those who complete and graduate, have ipso facto met state-established standards. 
However, these standards vary from state to state. Furthermore, no consensus has been reached 
about a practical definition of “industry-validated standards.” 

The National Skill Standards movement seeks to achieve a high level of state and local conver- 
gence in the routine recognition and use of common industry-validated skill standards. The 
National Skill Standards Board website (www.nssb.org) provides many examples of successful 
negotiation of such standards. 

Those who resist prodding to adopt standards of this type often hold that skill standards should 
be time and location sensitive. These stakeholders want local economic conditions and em- 
ployer personnel practices factored into skill standards decisions. 

The controversy over industry- validated skill standards recalls the contentious issue of perfor- 
mance indicator adjustment that was covered earlier. Pervasive recognition of local differences 
in labor market conditions and business personnel practices is inconsistent with adoption of 
just a few core indicators of performance. This is another example of what is given up. when ease 
of reporting is given a higher priority than possible validity and relmbility criteria applied to 
what is reported. 

Requirement of a level of skill standard achievement that is not in sync with state economic 
realities can precipitate unintended consequences of the performance measurement process. 
Some students might be denied admission who otherwise would have benefited from enroll- 
ment in a vocational and technical education curriculum. Some students might struggle to 
achieve national standards given current school investment decisions. Uncertainty about how 
these students will be treated in the calculation of subindicator performance levels can result in 
steps to discourage enrollment of these students. 

Some states will be exposed to public expressions of concern following the release of state com- 
parisons of core indicator definitions and values, if the media report on these comparisons. This 
will add momentum to “natural” convergence forces associated with work force mobility. The 
more the public knows about the performance measurement approaches of other states, the 
more vocal some state and local advocacy groups are expected to become in pressing for perfor- 
mance measurement refinements at home. 

The Definition and Measurement of 
‘^Secondary Credentiai Attainment” 

Another subindicator goal is that vocational and technical education completers receive a high 
school diploma or its state-recognized equivalent. A related subindicator goal is that these 
students also receive a proficiency credential that provides a more detailed record of their aca- 
demic and vocational- technical skill achievements. 
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Measurement issues associated with credential documentation have been covered in previous 
subsections. These are not repeated here. Suffice it to say that states differ in the sophistication 
of their credential recognition and documentation systems. The OVAE Peer Collaborative 
Network, which is designed to foster more frequent communication of state innovations in 
performance measurement, will broaden and deepen awareness of these differences. Whether 
and how states respond to this heightened awareness remains to be seen. 

The Definition and fleasurement of “Secondary Piacement” 

Core Indicator Three is intended to gauge a state’s progress in achieving high school student 
transitions to postsecondary education or advanced training, employment, or military service. 
A long list of performance measurement challenges emerged from this goal statement during 
the OVAE pilot test of state performance reporting capacity. 

• The concept of “successful transition” must be translated into practical data element defini- 
tion and time reference specifications. Currently, states differ in both definitions used and 
transition time references adopted. 

• There is a compelling need to establish a preferred sequence, or hierarchy, of placement 
destinations. Otherwise, a state can, in principle, report a single high percentage figure 
indicating that almost all vocational and technical education students who received a high 
school diploma or its equivalent in a reporting cycle had enrolled in postsecondary educa- 
tion, were employed, or had enlisted in the military. The Core Indicator cannot be changed, 
for now, but OVAE and the states can collaborate in the development and adoption of 
additional descriptors of these transition destinations, including insights about combina- 
tions of these transitory destinations. 

• It is reasonable to believe that members of Congress, their staffs, and the public assume that 
virtually all secondary vocational and technical education completers who receive a high 
school diploma or its equivalent will make the transition to a job or enroll in postsecondary 
education or advanced training. Minimal satisfaction of any one of these transition goals 
cannot be considered a serious outcome achievement without more information about the 
qualitative attributes of the employment, postsecondary education, or advanced training 
status in question 



• A growing number of states are using administrative records to document placement, but 
others continue to rely on follow-up surveys of differing quality. Administrative record cov- 
erage limitations are understood. Questionnaire design, sample specification, and survey 
administration deficiencies are also well documented. Neither data collection approach is 
perfect. Awareness of how to improve the quality of each method is growing. 
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Anderberg and Pfeiffer (1998) and Stevens and Shi (1996) cover many of the conceptual and 
practical challenges that are encountered when administrative records are used to document 
employment status. Stevens (forthcoming) provides detailed legal definitions of the basic cat- 
egories of employment that are not found in State Unemployment Insurance Wage Records. 
Also see the Florida Workforce Education and Outcome Information Services unit website 
(http://www.fim.edu/doe/weois/index.htm). 
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The most important of these omitted types of employment is independent contractors. U.S. 
Department of Labor’s (2000) Independent Contractors: Prevalence and Implications for Un' 
employment Insurance Programs is a useful reference source. 

Another important category of employment that is not found in State Unemployment Insur- 
ance Wage Records is federal government civilian employees and military personnel. The ab- 
sence of a single national gateway for state access to federal civilian employee and military 
personnel data remains frustrating and expensive. This issue continues to be negotiated among 
federal agencies that are responsible for potentially impacted data collection systems. For ex- 
ample, the Administration for Children and Families, Office of Child Support Enforcement, 
Department of Health and Human Services manages an Expanded Federal Parent Locator 
Service that includes a National Directory of New Hires. 

The National Directory of New Hires contains three types of information: 

1. Employment data on newly hired employees submitted by State Directories of New Hires 
and by federal agencies 

2. Quarterly earnings information obtained from state unemployment insurance agencies 
and from federal agencies 

3. Unemployment compensation claims data provided by state unemployment insurance 
agencies 

The purpose of the National Directory of New Hires is to be able to find information quickly 
about parents with child support obligations who are residing or working in other states. This 
expanded electronic file builds on years of state experience in conducting cross-matches be- 
tween state child support locate requests and State Employment Security Agency data files — 
the original Federal Parent Locator Service initiative that began in 1990. Beginning in October 
2001, the National Directory of New Hires will be used to calculate the employment and earn- 
ings performance indicator values for High Performance Bonus award purposes for the Tempo- 
rary Assistance for Needy Families program under the Personal Responsibility and Work Opportu- 
nity Reconciliation Act of 1 996. This precedent for expanded use of the National Directory of 
New Hires has renewed requests from other federal agencies, including the Department of 
Education and the Department of Labor, to be granted similar access for cost-effective calcula- 
tion of required core indicators of performance. 

State collection of out-of-state employment information is currently haphazard. The quality of 
out-of-state coverage in locally initiated student follow-up surveys is uneven. States that use 
State Unemployment Insurance Wage Records typically request access to their own state’s data 
only. 

Some states have entered into regional UI wage record sharing agreements that add limited 
coverage beyond a state’s own borders. There is active discussion among federal agencies con- 
cerning possible future access to the Wage Record Interchange System (WRIS) hosted by the 
National Association of State Workforce Agencies (NASWA). The NASWA website 
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(www.naswa.org) is a useful source of up-to-date information about the WRIS; choose “WRIS 
Watch” from the topic menu. 



OVAE must exercise careful oversight of state and local efforts to find as many former career 
and technical education students as possible, so they can be included in the numerator of the 
“placement in employment” subindicator under Core Indicator Three. 

First-time expanded coverage of data sources is expected to find new subpopulations of former 
students, such as by adding interstate matches of State Unemployment Insurance Wage Records 
or carrying out a match against Office of Personnel Management records to locate former stu- 
dents who are federal civilian employees. This will look like performance improvement in a 
year-to-year comparison, unless care is exercised to require the same specification of data sources 
in the benchmark and current reference years. 

Combinations of data sources that cannot be linked using a common identifier are likely to 
result in some duplication of “placement in employment” reporting. State Unemployment In- 
surance Wage Records are linked to student records using a social security number identifier. 
Survey results may not contain this identifier. Therefore, a summing of these two sources of 
“placement in employment” information would be expected to result in duplication in some 
cases. Selected audits of such circumstances, using other identifying information, can help to 
estimate the importance of this measurement problem. 



OVAE staff, and their Employment and Training Administration counterparts in the Depart- 
ment of Labor, can conduct or commission studies of the year-to-year variation in record match 
rates found in particular data sources. If a high level of stability in match rates is found, this 
evidence might be used to require less-than-annual data collection using some sources of “place- 
ment in employment” information. 

For example, if the percentage of former students in a reference cohort who are found in the 
Unemployment Insurance Wage Rates maintained by other states remains essentially constant 
over time, then this, percentage might be transformed into a cost-effective adjustment factor 
that can be used for, say, 3 years, instead of encouraging annual collection from this source. 

The effort that is made to find “everyone,” and the costs associated with such an intensive 
effort, ultimately must be justified on the basis of what this effort contributes to customer needs 
for performance indicator information. One expected return on investment is that a high “place- 
ment in employment” figure will lower the probability of an unfavorable comparison with other 
states. However, offsetting this presumed benefit is awareness that a high “placement in em- 
ployment” figure this year becomes the benchmark for calculating future continual progress. 
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Sampling remains a necessity in some states, when student social security numbers are not 
available to be used for linkage with State Unemployment Insurance Wage Records, or when 
these records are not accessible for other reasons. There is a new criterion to consider when 
sampling designs are developed. Linkage of secondary and postsecondary data files, and interest 
in linkage of these with WIA Title I and Title II participant records, introduces a longitudinal 
perspective to data collection and maintenance. This time horizon is a serious challenge to the 
ease of selecting a sample of students from a cohort of high school concentrators or completers. 
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Beyond core indicator reporting, interest has been expressed in tracking high school career and 
technical education students for years, as they flow in and out of postsecondary education and 
employment affiliations. An adequate sample size to satisfy immediate core indicator reporting 
is unlikely to suffice for long-term follow-up purposes. 

There is a tradeoff that has not been examined in most performance measurement forums. 
Evaluation funds can be spent in an attempt to find “everyone” or at least some of these funds 
can be redirected to development of a more detailed description of the employment circum- 
stances of a representative subpopulation of the reference cohort of former students. 

A high return on investment from redirected funds would be expected. Many of the customers 
identified in figure 1 (p. 25) want to know how former career and technical education students 
have fared with respect to challenging job assignments, promotion and continued learning 
opportunities, earnings gains over time, and benefits coverage. 

Ultimately, the convergence momentum sought by OVAE through its Core Indicator Frame- 
work and Peer Collaborative Network should be motivated by a federal-state consensus about 
what customers want to know about the employment of former vocational and technical edu- 
cation students. 

The Definition and Measurement of 
Postsecondary Core Indicators 

The issues covered in the previous seven subsections are just as applicable to postsecondary 
measurement as to the secondary indicators addressed. An important difference between sec- 
ondary and postsecondary is the latter’s challenge to identify a cohort of students for measure- 
ment purposes. The difference is one of degree only, because secondary students also enter and 
exit career and technical education courses in diverse patterns. 

The concepts of “placement in employment” and “retention in employment” are particularly 
limiting with respect to postsecondary performance measurement. Most postsecondary stu- 
dents work part time while enrolled. Some remain affiliated with one of these employers after 
leaving school. The issue of interest for many customers of performance data then becomes 
descriptors of the quality of employment, not just the status of being employed. 

Continuai improvement 

A key phrase in the ISO quality management principle of continual improvement is “establish- 
ing goals to guide, and measures to track, continual improvement.” This is a fundamental pur- 
pose for the OVAE Data Quality Initiative and Peer Collaborative Network, to negotiate quan- 
tifiable goals with each state and then hold the state accountable for valid and reliable mea- 
surement of progress toward these goals. 

, Measurement of continual improvement requires a basic level of stability or continuity in the 
management information systems relied upon to provide repeated measurement of agreed upon 
performance indicators. Changes in data element definitions, in the timing of data collection, 
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in the methods used to obtain data, and in quality control practices represent a potential threat 
to the integrity of continual progress calculations. 

OVAE staff and their state partners in Perkins 111 performance accountability, as well as ETA 
staff and their state partners, face a difficult challenge. Each team of partners is making a 
serious effort to achieve continual improvement of performance measurement. But changes in 
measurement practices can interfere with an accurate measurement of continual progress of 
performance itself There is a virtual certainty that the No Child Left Behind Act of 2001 and 
periodic changes in state performance indicator systems will combine to make it very difficult 
to track Perkins 111 performance indicator trends. 

Taking Action Based on Tactual Analysis 

The subheading is a phrase from the ISO quality management principle a factual approach to 
decision making. A similar phrase, “selection of indicators and evidence of their use and effec- 
tiveness in daily operations,” appears in the Baldrige National Quality Program Education cri- 
terion “information and analysis.” Each of these criteria of quality management takes one step 
beyond the OVAE Data Quality Initiative. The OVAE Initiative concentrates on continual 
improvement of performance indicator definitions, data collection practices and quality con- 
trol steps. The linkage of better performance data with better management of vocational and 
technical education is a basic component of the Peer Collaborative Network and the technical 
assistance forums convened by OVAE. However, collection and dissemination of evidence of 
use and effectiveness in daily operations are beyond the explicit scope of the Data Quality 
Initiative per se. 

It is important to highlight another phrase from the ISO principle a factual approach to deci- 
sion making — “making decisions and taking action based on factual analysis, balanced with 
experience and intuition.” Quality management does not emerge from unquestioning allegiance 
to quantitative performance data alone. This is one reason why the core indicators are de- 
scribed as minimal required evidence of performance. 

The Remaining Quality Management Principles 



Six ISO quality management principles have not been covered to this point — customer focus, 
leadership, involvement of people, process approach, system approach to management, and 
mutually beneficial supplier relationships. The focus of each of these principles has been touched 
upon throughout this chapter’s description of the OVAE Data Quality Initiative. 



There has been a direct “customer focus,” although this has highlighted the requirements for 
state reporting to the U.S. Department of Education, with relatively little attention by the 
OVAE-state partnership to other customer needs and preferences for data definitions, quality, 
timing of availability, format, and cost. These considerations are not part of the mission of the 
federal-state partnership. Having said this, the Peer Collaborative Network established by OVAE 
and the states is designed to promote frequent sharing of state and local successes in responding 
to customer needs for performance information. 
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OVAE, some states, and some stakeholder representatives have exercised leadership in the 
implementation of the Perkins III accountability systems. Another phase of leadership will be 
required when the consequences of performance indicator availability are expected to occur. 

The OVAE'State partnership, the forums hosted by OVAE, and the Peer Collaborative Net- 
work demonstrate a commitment to the involvement of people criterion for quality manage- 
ment. OVAE and state actions with regard to the “process” and “systems” approach criteria for 
quality management are described in detail in this chapter. 

The “mutually beneficial supplier relationships” criterion for quality management refers to a 
quid pro quo environment in which OVAE requires certain actions from the states to meet its 
reporting responsibilities, and the states, in turn, express their needs and constraints in collect- 
ing performance information. 

Beyond the Core Indicators 

Beginning in the mid-1990s. Jay Pfeiffer, Chief of the Bureau of Workforce Education and Out- 
come Information Services in the Florida Department of Education, began to describe his re- 
sponsibilities as falling into three tiers of performance measurement: 

1. Tier One is composed of only the federal core indicators, which remain in parallel silos 
labeled Perkins III, WIA Title I, WIA Title II, WIA Title IV, TANF, etc. 

2. Tier Two includes these core indicators, complemented by other state-specific required 
indicators. 

3. Tier Three includes the federal and state required indicators, plus other state and local 
discretionary indicators. 

This has come to be known as the wedding cake visualization of performance measurement, 
because Tier Three includes more indicators than Tier Two, which in turn includes more indi- 
cators than Tier One. The robust set of required and discretionary indicators found in Tier 
Three is a necessary solid foundation that supports the lighter (i.e., more limited) upper tiers. 

The ISO quality management principle “taking action based on factual analysis” focuses atten- 
tion on Tier Three indicators. From a state and local perspective, collection of the information 
needed to enter denominator and numerator values for core indicator calculations is a Tier 
One exercise undertaken for someone else — OVAE. Local satisfaction of state reporting re- 
quirements can be described in the same way, as a Tier Two data collection and processing 
activity undertaken for someone else — the State Department of Education. Data elements that 
appear only in Tier Three are not there to satisfy someone else’s information needs. These are 
optional, or discretionary. They can be added or deleted, depending upon state and local needs. 

Employment of former students is used here to illustrate the three -tier approach to collection 
and use of performance information. 
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Tier One 



The core indicator is “placement in employment.” The numerator value, employment, might 
be measured by use of State Unemployment Insurance Wage Records, if student social security 
numbers are available, or through use of a mailed or telephone survey instrument. All that is 
needed is documentation that a former student was employed during a specified time inter- 
val — say, during the second quarter following the former student’s reference date of completion 
(or graduation). 

If a student’s reference date is June 2001, the follow-up quarter would be 2001:4, or the Octo- 
ber-December 2001 quarter. If the former student’s social security number is found in the state’s 
Unemployment Insurance Wage Record file for the 2001:4 quarter s/he is counted as employed 
in the numerator of the Core Indicator Three value. Or, if a former student self-reports that s/he 
was employed during the reference months in responding to a mailed or telephone survey ques- 
tion, s/he is counted as employed. 

The Tier One federal reporting requirement has been met. Documentation of “placement in 
employment” has been achieved. 

Tier Two 



Some states still use “placement in training-related employment” as a performance indicator. 
The training-related part of the definition goes beyond the federal “placement in employment” 
requirement. This information cannot be obtained from State Unemployment Insurance Wage 
Records (except in Alaska). 

There is no consensus about the definition of “training-related employment.” My view is that 
changes in business personnel practices and enrollment patterns in career and technical educa- 
tion combine to render the concept of training-related employment unreliable. 

Narrowly defined occupations with a clear specification of incumbent responsibilities have, in 
many cases, given way to broad definitions that reflect expectations of on-demand problem 
detection and resolution. At the same time, movement of a cohort of students through an 
identical curriculum has been replaced by multiple paths reflecting different student objectives. 
Together, these trends have resulted in greater diversity of preparation and less similarity of 
employment circumstances. 

Some states have substituted an industry code for occupational information because industry 
information is available from a State Employment Security Agency. This approach is challeng- 
ing for multiple reasons. 

1. The industry code is usually not found in the state agency’s Unemployment Insurance 
Wage Record file, so a request for this information increases the burden and cost of 
satisfying the request. 

2. States are now in the process of switching from long-standing use of a Standard Industrial 
^ ^ Classification (SIC) coding taxonomy to a new North American Industry Classification 
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System (NAICS) taxonomy. The NAICS substitutes a five-digit coding scheme for the 
SIC four-digit format. SIC major group code 73, Business Services, is split into multiple 
codes in the NAICS format. SIC major group code 58, eating and drinking places, which 
was in the retail trade division, will no longer be in retail trade. 

These, and other, changes must be understood by states that have been using the SIC cod- 
ing format. From a value-added perspective, what customer need does the resulting infor- 
mation satisfy? 

3. The connection between industry and evidence of training-related placement may be 
tenuous. Consider SIC Division I, Services; Major Group 80, Health Services; Industry 
Group code 805, Nursing and Personal Care Facilities; Industry 8059, nursing and per- 
sonal care facilities not elsewhere classified. Assume that a former student completed a 
career and technical education program under the 1990 Classification of Instructional 
Programs (CIP) Series 51, health professions and related sciences; code 51.07, health 
and medical administrative services. 

What level of SIC and CIP coding should be selected to decide whether employment is 
training related? The SIC Division level — Services— is too high a level of aggregation, and 
the four-digit level, nursing and personal care facilities not elsewhere classified, is much too 
detailed for this intended use. At the two- or three-digit level of SIC coding, there is no 
direct link between employment in a nursing and personal care facility and confirmed use of 
skills learned in a health and medical administrative services program. 

The health services example was chosen because there is a high probability of training- 
related relevance. But many other possible pairings of SIC and CIP codes are less straightfor- 
ward. 

The criterion for relevance used by one state is if a relevant Occupational Employment 
Statistics (OES) code appears in the staffing pattern of an industry. The OES code is used by 
a State Employment Security Agency to collect occupational staffing information from em- 
ployers. 

In this case, a former student’s actual occupational assignment is not known. An expedient 
assumption is made. If some reporting firms in the retail trade industry employ electricians 
and a former student has completed a program in CIP code 47.01, electrical and electronics 
equipment installers and repairers, and that former student is found in State Employment 
Security Agency administrative records as employed in the retail trade industry sector (SIC 
codes 52-59), then this will be considered a training-related placement. 

Another state has used historical data and past training-related employment assignment deci- 
sions to develop a crosswalk between career and technical education programs and occupa- 
tional information obtained from employers who hired completers of these programs. This soft- 
ware handles routine traditional occupations that have been reported in the past, but new 
occupational titles must be assigned a code and then a training-related decision must be made. 
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Regardless of the decision-making criteria adopted by a state, the underlying issue should be 
how to define validity and reliability for a training-related employment indicator. Why is the 
customer asking for this descriptor? There are two likely reasons when the customer is a state 
department of education: 

1. The descriptor is a core indicator defined in state legislation. 

2. The descriptor is thought by the department staff to be a valid indicator of a former 
student’s use of the academic and technical skills, providing evidence of an immediate 
payoff on the investment made to provide the educational opportunity. 

A different threshold level of measurement quality applies to each of these purposes. The core 
indicator requirement in state law might be satisfied by relatively inexpensive, but reliable, 
sample data, not necessarily collected every year. The evidence of an immediate payoff criterion 
requires more detail if the intended use of the information is to gauge whether the academic 
and technical skills offered are appropriate and sufficient. 

The statutory requirement provides legislative advocates with performance information that is 
needed to promote continued and higher levels of investment in career and technical educa- 
tion. The strategic management use of information supports routine monitoring of the align- 
ment of program offerings with employer expectations. 

An important conclusion at this point is that a higher level of measurement quality is required 
for strategic management uses of performance information than for core indicator use of the 
data. The core indicator is a macro, or broad-brush use, whereas strategic management deci- 
sions are micro, or special purpose motivations for information use. 

The conclusion reached about different demands for performance measurement precision is of 
particular importance with regard to the “placement in employment” indicator. Recall that the 
purpose of Perkins III is “to develop more fully the academic, vocational, and technical skills of 
secondary students and postsecondary students who elect to enroll in vocational and technical 
programs.” 

Core Indicator One (academic and technical skill attainment) and Core Indicator Tivo (diploma, 
credential and degree attainment) are direct gauges of success in the pursuit of continual progress 
in the development of these skills. Core Indicator Three (placement in further education, ad- 
vanced training, the military and employment) is an indirect measure of the use of these skills. 

The Perkins III conclusion does not apply to the core indicators for WLA Title I and Title II. 
The purpose of WLA Title I is “to provide workforce investment activities... that increase the 
employment, retention, and earnings of participants.” 

Employment is a direct indicator of WIA Title I performance. Similarly, the purpose of WIA 
Title II is “to assist adults to become literate and obtain the knowledge and skills necessary for 
employment and self-sufficiency.” 



ERIC 



54 



61 




Processes 



This casts employment as a more direct indicator of performance than the Perkins III Core 
Indicator Three, but less direct than the WIA Title II core indicator. 

Ultimately, an important performance measurement system design question becomes: How much 
do I want to invest in a particular indicator to achieve a desired level of quality (i.e., reliability) 
with respect to a customer’s need for performance information? 

Tier Three 

The Tier One and Tier Two indicators must be prepared. The customers for these are federal 
and state agencies. The indicators are required by law or by administrative edict. Definitions for 
these indicators are given to data collectors and processors. Measurement quality standards are 
negotiated with the superior authority. Tier Three responsibilities are totally discretionary and 
remain at the state and local levels. 

Typical customers for Tier Three information include lay members of school boards, constitu- 
ent advocacy groups that often represent particular gender and ethnicity interests, media rep- 
resentatives, parents, and students. Their questions, individually and collectively, focus on de- 
scriptors of employment quality. Employment status perse is relevant — do completers of a given 
career and technical education program find jobs? And training-related employment might also 
be of interest — -if my daughter completes this career and technical education program, can she 
expect to find a job using the new knowledge and skills? 

But earnings profiles over time, stability of employment opportunity, promotion paths and prob- 
abilities, descriptors of compensation packages including benefits received, and job satisfaction, 
for particular subpopulations, are of equal or greater importance to satisfy customer requests for 
performance information. 

Reluctance to invest in collection of these kinds of employment descriptors is expressed by 
those who think that educational performance responsibility should end at the point of knowl- 
edge and skill attainment, not subsequent use of these new competencies, which is subject to 
forces over which the educators exercise little control. The purpose stated in Perkins III lends 
support to this point of view. But this has no bearing on the flow of questions from customers. 
Their interests in performance information are not limited to the purpose of the act. 

The OVAE Peer Collaborative Network is one attractive medium for the exchange of informa- 
tion about Tier Three data collection activities. Some of the customer questions to be answered 
are easily described: 

• How have the young and sometimes not so young people who completed a secondary or 
postsecondary career and technical education program fared' in the workplace? 

• What distinguishes the program completers who “make it” in the world of work from 
those who do not succeed? 
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• What pairings of secondary academic and technical skill preparation with postsecondary 
academic and technical skill acquistion offer particularly attractive opportunities for stu- 
dents? 

Caution is urged in responding to appeals that persistent labor shortages exist in particular 
occupations. This is a complex issue that cannot be adequately addressed here, but the cau- 
tionary warning is expressed because there are many definitions of a labor shortage, clear evi- 
dence of a shortage is often missing, others may also be responding to an alleged shortage, and 
many forces can eliminate an imbalance before an educational response plays out. 

Opportunities to improve performance through adoption of improved performance measure- 
ment practices are most likely to emerge from increased attention to Tier Three discretionary 
indicators. These are the indicators that are most likely to translate into strategic action re- 
sponses. 

One exception to this strong statement is state performance -based funding that depends, at 
least in part, upon core indicator values. However, the recent history of the use of such funding 
criteria suggests that it is difficult to sustain a critical mass of political support for reliance on 
quantitative performance information. The availability of reliable performance information re- 
duces the power of those who advocate resource allocation based on other criteria. 

What, then, can be said about the future of vocational and technical education and work force 
development accountability? How might 21st-century accountability differ from the 20th-cen- 
tury practices described in chapters 3 and 4? Answers to these questions are offered in chapter 
5. 
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TTiis concluding chapter returns to four basic criteria for high-quality outcome -based perfor- 
mance measurement: 



1. A clear goal statement 

2. Valid and reliable measurement 

3. Standards adjusted for “relevant” differences in circumstances 

4. Performance -based consequences 

A fundamental requirement before undertaking the design of a high-quality performance mea- 
surement system is to ensure that there is a consensus that a practical capability exists to con- 
nect reliable measures of program activities to reliable indicators of performance. Stakeholders 
can disagree about program goals and performance indicators. All that is needed is agreement 
that relevant program activities and reliable performance measures can be linked in a practical 
way. 

A Clear Goal Statement 



The layers of management responsibility and data collection and processing illustrated in figure 
1 highlight the unavoidable fact that different customers for performance information are likely 
to focus on different goals for vocational and technical education and work force development. 

A particular module of a performance measurement system can be designed to address each 
goal. Overlap and definitional frictions among modules can be identified and reconciled in the 
design activity. This is not a one-time requirement at the outset. Module redesign and recon- 
ciliation will be necessary each time new customer needs are identified and accepted as a per- 
formance measurement system responsibility. 

Valid and Reliable Measurement II 



A short list of core indicators is unlikely to satisfy a diverse population of performance informa- 
tion customers. Consensus about validity is less likely to be reached about macro indicators, 
such as academic and technical skill attainment, or placement, than about micro indicators, 
such as student attainment of industry-validated skill standards for a particular career and tech- 
nical education program offered in a specific location at a designated time. 



Lags in the availability of some performance information, an absence of benchmark data for 
some measures, and the cost and administrative burden of establishing and maintaining linked 
longitudinal files limit the practical feasibility of adopting some preferred valid indicators of 
performance and aggressively pursuing professional standards of measurement reliability. 



A particularly difficult challenge faced in the refinement of performance measurement systems 
is the need to define before and after points in time that are consistent with the validity and 
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reliability criteria. Descriptors of “before” status are needed to calibrate a performance mea- 
surement calculation— what relevant descriptors are needed to describe a student’s academic 
and technical skills before beginning a given career and technical education program? Descrip- 
tors of “after” are needed to place a former student’s academic and technical skills and skills use 
in a proper context of influences other than the given career and technical education program. 
These descriptions of before and after assume that there is a practical way to define and record 
the timing of program entry and exit. The OVAE-state partnership has defined participants, 
concentrators, completers, and completers who graduate as a practical way to define before and 
after. WIA Title I regulations define “hard” and “soft” exit dates; the former being event-trig- 
gered by a documented transition from program participation to another status, whereas the 
latter is defined by the absence of a relevant action within a designated time period. 



The need to report to a higher authority within a short period of time limits a program manager’s 
flexibility in selecting and defining valid and reliable performance indicators. The Secretaries of 
Education and Labor must report to Congress annually, and each faces reauthorization delib- 
erations in 2002 and 2003. Some states face similar reporting and legislative sunset circum- 
stances. These reporting schedules and cycles threaten the consensus-building process among 
levels of management responsibility and data collection and processing. There are serious con- 
straints on federal, state and local capabilities to link valid and reliable before, during, and after 
measures of performance for programs affected by 1998 legislation. Reliable linkage of action 
and consequence is difficult, even without the added burden of an artificial criterion of annual 
reporting and transmittal of comprehensive assessment information by mid- 2002. 



A tension cannot be avoided between continual improvement of performance measurement 
and reliable measurement of continual improvement of performance. Reliable documentation 
of performance improvement requires before and after measurement using the same indicator 
definitions. Refinement of definitions, or changes in the quality of performance information 
collected, drives a wedge between the before and after measurement values. 

II Standards AcUusted for “Relevant” Differences 



Core indicators of performance are intended to be transparent, i.e., easily and correctly under- 
stood by a nonexpert. Any adjustment of core indicators compromises this criterion. Adjust- 
ments are needed for equity of treatment reasons. Therefore, the design of a Tier Three module 
for a comprehensive performance information system should include the definition of adjust- 
ment factors. These should not appear in the Tier One and Tier Two modules, where transpar- 
ency and conformity with federal and state definitions is required. The availability of adjust- 
ment indicators in Tier Three enables those who are interested to compare raw core indicator 
calculations with adjusted indicator values. The findings from this comparison can then be 
used for advocacy and negotiation purposes. 

II Performance-based Consequences 



o 
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Public disclosure of core indicator information, without adjustments, will be an unstable justi- 
fication for outcome-based actions to reward or punish program managers. The availability of 
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complementary state and local Tier Three descriptors will improve the quality of federal-state 
and state-local negotiations in the vocational and technical education and work force develop- 
ment spheres. 



The future of 
Performance 
Measurement 



Federal and state authorities have exhibited an understandable reluctance to base funds alloca- 
tion decisions on quantitative evidence of outcomes-based performance. The reluctance is un- 
derstood because the core indicator framework is meager. States such as Florida that have 
taken a bold approach to performance -based funding of some components of the state’s portfo- 
lio of education programs are experiencing the inevitable tension between performance -driven 
allocation criteria and the expression of other interests by parties that cannot be shut out of the 
process. 



The availability of reliable Tier Three descriptors increases the management leverage main- 
tained by education and work force development authorities. This assumes, of course, that 
there is a general consensus about the appropriateness of the descriptors, i.e., that they are valid 
measures of performance. 



The Reality of Devolution 



The nation’s vocational and technical education systems are different in many fiscal and man- 
agement authority respects. However, these systems are similar in one important respect — each 
depends upon local and state collection of performance information. There has been more top- 
down control of the WIA performance measurement process to date than is observed in re- 
viewing the OVAE-state performance measurement partnership activities. Nevertheless, each 
of these hierarchies of data collection and processing incorporates a healthy recognition that 
common definitions of performance indicators do not require uniform management information 
systems in each locality. 



State and local performance information systems existed before Perkins III and WIA, and they 
will continue beyond the sunset dates of each of these acts. The vocational and technical edu- 
cation and work force development modules of these information systems must interact with 
other components of larger management information systems. Sometimes, changes in the vo- 
cational and technical education or work force development modules trigger necessary changes 
in the other components. More often, changes in other components have consequences for the 
ease, and even the feasibility, of sustaining the status quo in the vocational and technical edu- 
cation and work force development modules. The No Child Left Behind Act of 2001 is expected 
to be an immediate example of this influence. 



Changes in public attitudes about access to social security numbers can create or destroy oppor- 
tunities to use administrative data sources as a cost-effective way to collect reliable longitudinal 
performance information. These changes can be sudden and are almost always unrelated to 
vocational and technical education and work force development performance measurement 
issues. 
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A Final Thought 



Nearly a half-century ago, Arthur Ross, a University of California at Berkeley professor of in- 
dustrial relations, coined the phrase “orbits of coercive comparison” to describe how successful 
negotiation of attractive gains by one union often resulted in similar gains by other unions. The 
characterization of this comparison as coercive arose from recognition that the comparison 
could not be avoided or ignored. 



Orbits of coercive comparison are emerging among the states as WIA Consumer Reports Sys- 
tem information begins to flow to the public. Other orbits of comparison will become more 
apparent as state annual reports to OVAE are released. 



Unlike a “corrosive” effect, which describes deterioration or an eating-away process, a “coer- 
cive” effect is a healthy market phenomenon — awareness of a higher- quality performance mea- 
surement system elsewhere creates pressure to emulate, or better yet, leapfrog this new plateau 
of performance measurement quality. 
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